[Phronk]

Originally Posted by Rudism

This is equivalent to saying "the reason we threw out the results was because we wanted to see what happened with the results we didn't throw out." It doesn't answer the question why were they thrown out? I'll address this again later on.

Well, something worth mentioning before I go on is that even if you're right and they shouldn't have been thrown out (though I still disagree), the results are still significant if they are analyzed your way. So arguing this point doesn't change the conclusions.

Let's look at your question (what to expect by chance alone): without a baseline (which this study did not compute),

Again, chance is the baseline. If we had some empirical baseline of how many times the bird says certain words on a regular basis, that would complicate the whole experiment and introduce more potential biases. Maybe the bird says "flower" a lot at home, but when in a lab and under stress, tends to say "flower" a lot less. Thus, comparing to the old flower baseline no longer applies and we'd find a statistical difference that has nothing at all to do with communication between the parrot and his owner. In a properly conducted study, testing deviations from chance is a much cleaner test of the hypothesis with less potential for bias.

---

Note that for the following examples, you are assuming that multiple responses in a trial were kept, so that the bird could have more than one "guess" per trial. I did not make this assumption in my reasoning to keep it simple, so it doesn't really apply. However, I looked at the original study (http://www.scientificexploration.org...ke_morgana.pdf) and they did indeed allow more than one guess per trial. While this complicates things, it does not change the general point of my previous post, as I will demonstrate below.

I'll try to illustrate with another example... Let's pretend the bird continuously talks throughout the entire experiment, just saying "ball bike flower" over and over again non-stop. If you show a random picture of either a ball, a bicycle, or a flower to the owner, then record what the bird says during the next two minutes, he's going to get a hit 100% of the time, even by chance.

Yes. Exactly. So the chance baseline is now 100%. This is a special situation where it's impossible to test if the bird scored higher than the chance baseline. But add in a fourth picture to the randomly selected pictures though, and a score above 75% on trials where the bird always said "ball bike flower", and sometimes a fourth word, would be evidence of communication.

This should be proof enough that the bird's baseline speech patterns alone (regardless of any psychic phenomenon) can obviously affect the statistics. You need to take it into account before you can point to anything as a possible anomaly.

Yes, the bird's baseline speech "affects the statistics". It changes the baselines we need to compare to for each subset of responses. And yes, it needs to be taken into account.

Luckily, it was. As I suspected, I believe the reason this study used uncommon statistical techniques (randomized permutation and bootstrap analyses) is that these take this unusual situation into account. By randomizing the specific set of responses given, the computer creates the chance baseline itself. Because the randomized responses also match up with the bird's multiple-guess format, this is taken into account. Out of all the randomized orders of guesses, the actual number of hits (or more) obtained in the real study were extremely rare. In other words, if chance alone were operating, that number of hits would almost never occur. Thus, it's unlikely that chance alone was operating. (Note: Don't quote me on this, as I'm not a total expert on these techniques - but don't dispute me either unless you have reason to!)

Originally Posted by Phronk View Post
To further illustrate why it is fair to throw out the other 90 trials, let's assume that only chance is operating. The bird is just babbling and is not influenced by the owner's thoughts at all.

On the first trial, the bird says "ball". What are the chances that he's right? Well, 1/3, because there's a 1/3 chance that the computer randomly selected the ball picture for the owner.

Wrong... The chance that you'll get a ball picture is 1/3. The chance that the bird will say "ball" cannot be calculated without knowing how often the bird says ball in general. Maybe he says ball every 1 minute on the minute, [...] during a 2 minute period after showing the picture, there is a 100% chance that the bird will say "ball" during that trial. That means in your post analysis, regardless of what the other trials say, you are going to see a 100% hit rate on the ball pictures.

Right. But we're not only looking at ball pictures. If we were, we'd hit the 100% baseline ceiling like in the example from before. In my original example, with a set of 3 pictures, if the bird said "ball" every minute, he'd get 100% on the ball pictures, 0% on the flower pictures, and 0% on the bike pictures. Overall, he'd get 33.33%. Exactly chance, and correctly, not interpreted as any kind of communication. All without knowing anything about the bird's baseline utterances.

So no, I'm not wrong. You say "the chances that the bird will say ball cannot be calculated without knowing how often the bird says ball in general". While true, it's beside the point. We're interested in when the bird says "ball" AND the ball picture was randomly selected. And that is something we can model perfectly without knowing what the bird says in general.

You are absolutely right when saying that we want to compare the actual results to what we would expect by chance alone. The only problem is that the experiment did not figure out what to expect by chance alone. I think I've sufficiently demonstrated this with my previous examples.

And I think I've sufficiently demonstrated that they did figure out what to expect by chance alone. Particularly important to the real study (vs. my example) is the randomized permutation analysis. I suggest looking up more about this technique if anyone is unfamiliar with it. I suspect the critics of this experiment did not do this, and thus their criticism was not entirely convincing (to me, at least). There may be biases in it, but to discover them would require digging deeper into the statistics.

I absolutely agree with this statement. Along the exact same reasoning, trials during which the bird did not respond with words matching to the owner's "psychic" signal are evidence that the stimulus does not exist. But most of this data was thrown out, apparently using the very excuse that the bird did not appear to get any psychic messages during that trial! If that's not bias, I don't know what is.

This again. I've already shown that looking at the subset of key words cannot possibly bias the results toward above-chance scoring. Even if you think including every trial would answer a question more directly (a different question, but still a question I guess), the significant results in the subset are still strong evidence for communication.

Besides, looking at it your way ends up giving evidence for communication too.

It all depends on the hypothesis being tested. If your hypothesis is that there are psychic messages being sent from owner to bird, then throwing out trials that are clearly evidence to the contrary is obviously biased. If, on the other hand, your hypothesis is that an autistic child will respond to stimulus X using method A more often than method B, then it is safe to throw out the trials where the child does not respond at all to stimulus X, because it doesn't say anything one way or another about the hypothesis (which is only interested in comparing responses).

See the difference? In the first case, every trial has important data that can answer the question being tested. We want to know, by looking at the bird's responses, if he's getting the stimulus at all. In the second case, only the trials where the child actually responds using either method A or method B are important to the question. If you were to do a study that hypothesizes autistic children will respond at all to stimulus X, then obviously throwing out trials where the child doesn't respond would be a ridiculous thing to do--just as it is in the parrot experiment.

Let's do a thought experiment.

If the bird experiment had added another variable, Method, like you propose for autistic children above, and found a difference in hit rates depending on method, would you then accept its results?

E.g. we find that when the owner is happy, the hit rate for happy pictures is higher than when the owner is sad, when the hit rate for sad pictures is higher. We throw out trials when the bird says things that are not related to either happy nor sad pictures.

We find a difference - the bird does better on happy pictures when the owner is sad, and the same for sad pictures and sad mood.

Do you say that this doesn't provide any evidence at all that the bird was reacting to the stimuli (the owner's photographs)? Was it biased because we threw out neutral words?

No. Of course not. If the bird is reacting differently to different "methods" (mood states), then he is certainly reacting! Throwing out neutral words made sense because they were irrelevant to the question at hand. But they didn't bias the results either way. The actual study done was even simpler, not including the extra variable, but threw out trials for the same type of reason.

In other words, if it doesn't bias results, and doesn't affect your conclusion, for autistic kids, why should it for parrots?