A couple of points:panda2rom wrote:I do agree with jkenny :)
And to be absolutly back to topics : trust your ears.
but JUST your ears.
It's pretty obvious visual will lead to bias : i won't talk about cable or small stuff. If you see me turning on a Mcintosh amp VS a plain looking sony, you will be biased.
There is actually a very serious study made on that subject ! Even trained ears are subject to bias, not just the newbs
The most stupid/common bias is about volume.
The louder always win.
So you have to be very carefull about that when doing testing.
Here is the study:
http://www.aes.org/e-lib/browse.cfm?elib=6338
And from Roger Russel webiste (that you all should read :P) :
"Floyd Toole presented a paper at the 97th convention of the Audio Engineering Society, November, 1994 titled Hearing is Believing vs. Believing: Blind vs. Sighted Listening Tests, and Other Interesting Things. Floyd originally worked at the National Research Council in Ottawa, Canada and then went to Harman International Industries, Inc., Northridge, CA. Floyd concludes: “Overall, though, it was clear that the psychological factor of simply revealing the identities of the products altered the preference ratings by amounts that were comparable with any physical factor examined in these tests, including the differences between the products themselves. That an effect of this kind should be observed is not remarkable, nor is it unexpected. What is surprising is that the effect is so strong, and that it applies about equally to experienced and inexperienced listeners.
Since all of this is independent of the sounds arriving at the listeners’ ears, we are led to conclude that, under the circumstances, believing is hearing, The bottom line: if you want to know how a loudspeaker truly sounds, you would be well advised to do the listening tests “blind.”
- yes, knowledge of what is being listened to CAN bias the results but I don't believe it necessarily ALWAYS does bias the result & I don't know if there are any studies which actually try to determine the strength of this bias in ordinary, normal listeners who have no skin in the game - that Harmon study is not exactly what I would call a study that tries to do this. Let's see - if we wanted to ascertain the strength that sighted bias has in ordinary people would we choose workers from Harmon ( a speaker manufacturer) to do the test on when we are testing speakers - some of which are Harmon speakers. Do you think the expectation bias of these Harmon workers might be a lot different than non-Harmon people - particularly when the test is being run in Harmon labs by a newly appointed Harmon researcher?
- The guidelines (BS.1116) for conducting a rigorous blind test are designed to try to minimise ALL biases, not just sighted biases - this is one of the big misunderstandings of home run blind testing & why my criticisms are directed towards these. These biases mostly skew the test towards returning a result which is no better than random I.e no difference heard. I nearly always see it stated on audio forums that eliminating sighted bias is "obviously better". Well, this is a fallacy.
Just because people can't hear what they could differentiate in sighted listening doesn't mean the test is "better" - it can simply mean that the test itself, without attending to these biases, masks most differences, except for gross differences. To evaluate if it is a better test we need to examine the sensitivity of the test - how good is it at revealing a difference that REALLY exists Vs how good is it at revealing that there is no difference when no differences exist. These are called false positives & false negatives & the level of each is needed to determine the sensitivity of the test. We don't have any statistics for false negatives for home run blind tests so we are left with not knowing the sensitivity of the test - we don't know to what level the test masks hearing real differences that actually exist between two things.
- The guidelines weren't created just for fun - they are there because it is recognised that lots of factors skew the results of perceptual testing & in order to achieve something approaching a valid test it's required that these factors are attended to. Yes, volume matching is one but also using equipments suitable for revealing the difference & one of the most important - training which is the most overlooked of them all. Because of the nature of blind testing, unless you have found a specific difference & can routinely identify it 9 out of 10 times, unless you can do this then you are almost certainly going to do no better than guessing in a blind test. Blind tests are mostly premised on the conscious identification of differences, seldom, if ever, are they run on the basis of how one feels overall
Anyway, enough of this. I believe home based blind testing has become a tool used by a particular mindset because it nicely returns mostly null results & has a soundbite which is believable to the unwary "removing sighted bias, ears only listening, must be better"