Re: Public 128 kbit/s Extension Test

[tony sayer]

In article >, ff123
> writes
>Roberto Amorim announced in various web forums:
>
>"I'd like to announce the start of a public 128kbps blind listening
>test, comparing the winner of the AAC test (QuickTime) to Musepack,
>Vorbis, WMA Pro and Lame MP3.
>
>For those interested in participating, instructions are available at
>the announcement page:
>http://rarewares.hydrogenaudio.org/test/128extension/presentation.html
>
>If questions or issues arise, please post at this thread."
>
>
>
>Here are the results from Roberto's previous, AAC-only test:
>
>http://rarewares.hydrogenaudio.org/test/aac128test/results.html
>
>ff123

You ought post this to alt.radio.digital and perhaps uk.tech.broadcast
--
Tony Sayer

[Jim H]

ff123 in uk.rec.audio:

> On Thu, 24 Jul 2003 19:34:13 +0000 (UTC), Jim H
> > wrote:
>
>>Have you considered that disclosing the encoders uesd may have a
>>subtle effect on the results? For example, open-source enthusiasts
>>might ignore vorbish artifacts.
>>
>>It would be interesting to look at how user prejudices alter
>>perception of quality. For example, to see if Linux users (who know
>>the samples' format) vote WMA down.
>
> That's why a blind testing utility is used:
>
> http://ff123.net/abchr/abchr.html
>
> ff123
>

For win32 users, yes. For users of other operating systems there is no
blind testing, that's why I specified Linux users, who are not tested
blindly but invited to comment on the quality of the files.

I'd be nice to look if, say, 50% of blindly tested users preferd wma,
whereas the for non-blind Linux users the positive attitude towards open
standards lowered the perception of wma quality.

Interesting study, btw. I'll be joining in.

--
Jim H
3.1415...4999999 and so on... Richard Feynman

[Jim H]

ff123 in uk.rec.audio:

> Roberto Amorim announced in various web forums:
>
> "I'd like to announce the start of a public 128kbps blind listening
> test, comparing the winner of the AAC test (QuickTime) to Musepack,
> Vorbis, WMA Pro and Lame MP3.
>
> For those interested in participating, instructions are available at
> the announcement page:
> http://rarewares.hydrogenaudio.org/test/128extension/presentation.html
>
> If questions or issues arise, please post at this thread."
>
>
>
> Here are the results from Roberto's previous, AAC-only test:
>
> http://rarewares.hydrogenaudio.org/test/aac128test/results.html
>
> ff123

The test isn't working for me. I've tried samples 2 and 9. when I select
the canfigXX.txt file I get an error dialogue "cannot open file .\sampleXX
\<name>_<codec>.wav", for each codec

--
Jim H

3.1415...4999999 and so on... Richard Feynman

[ff123]

On Thu, 24 Jul 2003 22:05:05 +0000 (UTC), Jim H
> wrote:

>The test isn't working for me. I've tried samples 2 and 9. when I select
>the canfigXX.txt file I get an error dialogue "cannot open file .\sampleXX
>\<name>_<codec>.wav", for each codec

You need to run the batch files first, which will create the WAV files
necessary for the test.

ff123

[Jim H]

Harri Mellin in uk.rec.audio:

> In article >,
> ff123 > wrote:
>
>> That's why a blind testing utility is used:
>>
>> http://ff123.net/abchr/abchr.html
>
> for Windoze only and windows users
>

Maybe writing the gui in java would have been a good idea, with native
codecs.

--
Jim H
3.1415...4999999 and so on... Richard Feynman

[ff123]

On Thu, 24 Jul 2003 21:36:00 +0000 (UTC), Jim H
> wrote:

>For win32 users, yes. For users of other operating systems there is no
>blind testing, that's why I specified Linux users, who are not tested
>blindly but invited to comment on the quality of the files.

Unfortunately, people who can't use the blind testing utility won't be
able to contribute anything more than comments. Any ratings they
provide can't be used towards determining the group preferences. The
comments might be useful for the codec developers, though.

ff123

[Arny Krueger]

"Jim H" > wrote in message

> ff123 in uk.rec.audio:
>
>> On Thu, 24 Jul 2003 19:34:13 +0000 (UTC), Jim H
>> > wrote:
>>
>>> Have you considered that disclosing the encoders uesd may have a
>>> subtle effect on the results? For example, open-source enthusiasts
>>> might ignore vorbish artifacts.
>>>
>>> It would be interesting to look at how user prejudices alter
>>> perception of quality. For example, to see if Linux users (who know
>>> the samples' format) vote WMA down.
>>
>> That's why a blind testing utility is used:
>>
>> http://ff123.net/abchr/abchr.html
>>
>> ff123

> For win32 users, yes. For users of other operating systems there is no
> blind testing, that's why I specified Linux users, who are not tested
> blindly but invited to comment on the quality of the files.

Actually there are blind testing programs for Linux and Mac users but they
don't exactly fit in with this particular test. Please see
http://www.pcabx.com/program/index.htm

> I'd be nice to look if, say, 50% of blindly tested users preferred wma,
> whereas the for non-blind Linux users the positive attitude towards
> open standards lowered the perception of wma quality.

The reason why there's a ABC/hr testing utility for Windows is that a
private individual wrote one on his own nickel. I believe that the source
code is readily available. I've heard a rumor that there are Linux users who
are capable of writing programs or at least porting Windows source code to
Linux.

[Arny Krueger]

"Jim H" > wrote in message

> Harri Mellin in uk.rec.audio:
>
>> In article >,
>> ff123 > wrote:
>>
>>> That's why a blind testing utility is used:
>>>
>>> http://ff123.net/abchr/abchr.html
>>
>> for Windoze only and windows users
>>
>
> Maybe writing the gui in java would have been a good idea, with native
> codecs.

For programs of this (modest) level of complexity, Java is way too slow on
most contemporary machines.

[Jim H]

Arny Krueger in uk.rec.audio:

> "Jim H" > wrote in message
>
>> Harri Mellin in uk.rec.audio:
>>
>>> In article >,
>>> ff123 > wrote:
>>>
>>>> That's why a blind testing utility is used:
>>>>
>>>> http://ff123.net/abchr/abchr.html
>>>
>>> for Windoze only and windows users
>>>
>>
>> Maybe writing the gui in java would have been a good idea, with
>> native codecs.
>
> For programs of this (modest) level of complexity, Java is way too
> slow on most contemporary machines.
>
>
>

I disagree. But java is, in general written very badly, swing particuarly
so.

There were a lot of big performance gains in the 1.2 reference virtual
machine and API implementation, for example reflection was sped up nearly
20x. With a JIT compiler the efficiency of java is very close to that of
the native platform.

All that is needed here is a very simple program. I don't want to be drawn
into a long OT discusion of programing languages but I'm certain java would
be acceptably fast on a recent VM.

--
Jim H
3.1415...4999999 and so on... Richard Feynman

[ff123]

On Thu, 24 Jul 2003 05:03:57 GMT, ff123 > wrote:

>Roberto Amorim announced in various web forums:
>
>"I'd like to announce the start of a public 128kbps blind listening
>test, comparing the winner of the AAC test (QuickTime) to Musepack,
>Vorbis, WMA Pro and Lame MP3.
>
>For those interested in participating, instructions are available at
>the announcement page:
>http://rarewares.hydrogenaudio.org/test/128extension/presentation.html

The test has closed and the results are now available here:

http://audio.ciara.us/test/128extension/results.html

The newer generation of codecs are all on a par with each other,
although still not transparent at 128 kbit/s. MP3 is showing its age.

ff123

[Ric]

So maybe the pertinent question now is: at the lowest quality setting for
each codec that produces a transparent file, which codec has the lowest
average bitrate?

Ric

"ff123" > wrote in message
...
> On Thu, 24 Jul 2003 05:03:57 GMT, ff123 > wrote:
>
> >Roberto Amorim announced in various web forums:
> >
> >"I'd like to announce the start of a public 128kbps blind listening
> >test, comparing the winner of the AAC test (QuickTime) to Musepack,
> >Vorbis, WMA Pro and Lame MP3.
> >
> >For those interested in participating, instructions are available at
> >the announcement page:
> >http://rarewares.hydrogenaudio.org/test/128extension/presentation.html
>
> The test has closed and the results are now available here:
>
> http://audio.ciara.us/test/128extension/results.html
>
> The newer generation of codecs are all on a par with each other,
> although still not transparent at 128 kbit/s. MP3 is showing its age.
>
> ff123