[Frank Stearns]

As time goes by, and the terabytes mount up, I'm getting
rather suspicious of USB-2 and NTFS (windows) file systems.

Spread over three computers, dozens of drives (SATA for use
with the DAW; IDE for use with the recorder), a couple of
external USB-2 housings, I keep getting annoying, random
errors at random intervals... Things such as a very short
burst of white noise in a wav file that was clean
originally; another few corrupted wav or fade files that
cause Protools to choke upon load (but all was well six
months ago); the occasional CD Architech file that now
claims it's not a CD arch file when it used to work just
fine; etc.

This only happens with something probably less than 0.001%
of the total byte count of data that's stored here, and I
can normally recover from a backup somewhere, but this
seems ridiculous and annoying. I sure as hell don't
remember UNIX or VAX/VMS systems doing this kind of thing,
but maybe the data sets just weren't large enough.

The common denominator seems to be USB, as the corruptions
tend to show up in files that have been moved via USB from,
say, an active drive to an archive drive. But I've also
seen NTFS misbehave and entire sets of files disappear then
reappear, sometimes on a refresh, sometimes only on a
reboot, and sometimes simply by moving through the folder
trees with explorer, even after multiple refreshes failed.

To copy files I generally use drag and drop via two XP
explorer windows; this supposedly automatically turns on
read-after-write verification, but perhaps I should only
use xcopy in the CMD window after having manually set
verify to ON. Perhaps verify-after-write is disabled across
USB connections (Windows trying to be "helpful" by
"speeding up" operations across a slower disk connection.)

I occasionally do other things with the system during such
transfers, perhaps foolishly believing that a quad-core
machine running XP SP3 can walk and chew gum at the same
time.

These are on fairly new, tweaked XP machines ("tweaked"
meaning they're not running a bunch of extraneous junk and
are favoring audio production).

I probably don't need to hear about Linux or the Mac; I'd run one of
the real UNIX machines running real UNIX if I could (and
did, once upon a time, in another life), but commodity
machines have put us where we are; that's the reality. I'm
looking for work-arounds.

Mostly curious about any patterns you've seen, assuming
you've had similar problems.

Thanks in advance,

Frank Stearns
Mobile Audio
--

[Richard Crowley]

"Frank Stearns" wrote ...
The common denominator seems to be USB, as the corruptions
tend to show up in files that have been moved via USB from,
say, an active drive to an archive drive.

I've been using USB2 to IDE cables and raw drives now for
years for both multi-track audio and video NLE. I have not
experienced that phenomenon.

OTOH, More and more computers feature external "eSATA"
connectors and 4-pin drive power connectors. I'm gradually
switching to direct eSATA as the IDE ("PATA") drives are
disappearing from the marketplace.
http://en.wikipedia.org/wiki/Serial_ATA#External_SATA

posting from Botosani, Romania on tour.

[david correia]

In article isition,
Frank Stearns wrote:

I probably don't need to hear about Linux or the Mac; I'd run one of
the real UNIX machines running real UNIX if I could (and
did, once upon a time, in another life), but commodity
machines have put us where we are; that's the reality. I'm
looking for work-arounds.




Not trying to get you switch your OS. Just wanted to ask why you think a
Mac running OS X isn't a "real UNIX machine running real UNIX"?


You may wanna check out: http://www.apple.com/macosx/technology/unix.html





David Correia
www.Celebrationsound.com

[Frank Stearns]

david correia writes:

In article isition,
Frank Stearns wrote:

I probably don't need to hear about Linux or the Mac; I'd run one of
the real UNIX machines running real UNIX if I could (and
did, once upon a time, in another life), but commodity
machines have put us where we are; that's the reality. I'm
looking for work-arounds.

Not trying to get you switch your OS. Just wanted to ask why you think a
Mac running OS X isn't a "real UNIX machine running real UNIX"?

You may wanna check out: http://www.apple.com/macosx/technology/unix.html

Interesting, and hopeful; but for a number of mundane reasons I'm stuck
using PCs.

Besides what seemed to be a robust file system, I also miss the 300-400
days between boots on my SPARC stations. Sigh.

Frank
--

[Frank Stearns]

"Soundhaspriority" writes:

- snips -

To copy files I generally use drag and drop via two XP
explorer windows; this supposedly automatically turns on
read-after-write verification,

I never heard of this feature. As far as I know, there is no automatic
read-after-write verification in XP, unless a command line utility is used.
Perhaps you are thinking of disc burning?

Most burning sw does indeed have a "verify after write" option. But no, I was
thinking specifically of regular copy operations within windows, disk to disk
(folder to folder).

I've had cause to investigate this deeply in the last month or so, and stumbled
across documentation from MS that supposedly "VERIFY" is switched on when dragging
and dropping via explorer. After all these years I'd never heard that, but thought,
well, if true, that's good; and maybe it explains why such copy ops can be rather
slow.

Frank, it's obviously impossible to do more than speculate. However, I will
tell you this is why I use only ECC memory in my machines. Since you are an
oldtimer, you are probably more aware than others that electrical noise in
computers follows a Gaussian model. Computers can't be divided
simplistically into machines that work, and machines that don't. In every
computer, there is a sea of noise sloshing around, as well as cosmic rays
that flip memory cells.

Good point. It just seems that as "small" as the error rate seems to be, a bank,
let's say, wouldn't stand for it -- ECC would help.

decreed that a consumer desktop doesn't need ECC. They took ECC support
completely out of the I7, requiring purchase of the Xeon 5500 equivalent at
6X the price if ECC is required. In marketing terms, this is Intel's
strategy for segmentation of the marketplace, something I disagree with,
because when my computer flips a bit, I don't give a damn about the other 99
people.

Ah. Another good point.

And this is all most interesting. I suspect that perhaps such errors become
statistically more apparent just given the sheer volume of data. Even five
years ago the idea that my little one-man shop would be pushing somewhere around 22
Tbytes would have brought a chuckle.

AMD has consistently made ECC available to the masses, so I chose slower AMD
processors, the Phenom II 940, and the peace of mind that my Asus 790FX
based motherboards actually have IBM's patented "chipkill feature", meaning
that even if a DRAM dies on the run, the machine just keeps going like
nothing happened.

But getting back to the issue of pure noise margins, the faster you go, the
higher the equivalent Gaussian noise power in the machine, and the greater
the chance of a freak wave flipping a bit. Underclock the machine, and the
reverse is true. "Speed kills" was never more true than with computers.

Hmmm. I might try that.

My experience with quad cores under XP is that they can behave oddly, but
never destructively. You may still have a bad component, a bad hard disk, or
a bad motherboard, or bad memory, yet the oddness of the way quad cores act
and feel may have obscured this to you. So I suggest, ignore the oddness,
and come to a certain conclusion about possible defective hardware. You may
wish to consider underclocking as a noise diagnostic.

Part of the frustration is never knowing when something will go/has gone south short
of spending more hours than there are in a day verifying after every operation. It
always shows up months and months later, after revisiting the project for whatever
reason.

You use it, you finish, all is well, yet there it sits, crumbling in small invisible
ways. Criminy. I do image to a second drive, and every so often to another backup
machine on the LAN, but perhaps it's time to also integrate the production of "first
set" DVDs immediately, rather than waiting, then also do another archive set to DVD
later. I am already sending critical stuff off-site to a commercial backup site over
the net. Sending ALL the data this way isn't feasible.

Tape was never like this... Oh wait, sticky shed! Bad caps! Oxidizing connectors!
(Oh my...)

Thanks for the input.

Frank
--

[Scott Dorsey]

david correia wrote:
Frank Stearns wrote:

I probably don't need to hear about Linux or the Mac; I'd run one of
the real UNIX machines running real UNIX if I could (and
did, once upon a time, in another life), but commodity
machines have put us where we are; that's the reality. I'm
looking for work-arounds.

Not trying to get you switch your OS. Just wanted to ask why you think a
Mac running OS X isn't a "real UNIX machine running real UNIX"?

Well.... secretly it's really Mach inside if you look deeply enough. I think
this is a good thing, better than a real Unix kernal in a lot of ways. On
the other hand, it almost meets gus baird's standard of having every system
call that V7 has.

I will say that Apple's implementation of USB is radically different than
the route every other Unix dialect decided to take. This has been a real
problem porting drivers for USB devices to and from OSX. I'm not saying that
the route they took is better or worse (and it has some philosophical
connection to the old Apple Desktop Bus implementation), but it's different.
--scott


--
"C'est un Nagra. C'est suisse, et tres, tres precis."

[Scott Dorsey]

Frank Stearns wrote:
Interesting, and hopeful; but for a number of mundane reasons I'm stuck
using PCs.

Besides what seemed to be a robust file system, I also miss the 300-400
days between boots on my SPARC stations. Sigh.

Actually, there's no reason you can't use an industrial-strength computer
for your file server and SMB the data off to client PCs running Windows.
Enough people have had a need to do this that there are even turnkey systems
available.
--scott

notavax% uptime
10:55AM up 342 days, 6:37, 81 users, load averages: 3.31, 3.75, 3.71

--
"C'est un Nagra. C'est suisse, et tres, tres precis."

[Frank Stearns]

"Soundhaspriority" writes:


"Frank Stearns" wrote in message
ion...
"Soundhaspriority" writes:


- snips -


If it's online, would you be so kind as to post a link? I did a search, but
didn't find it.

Darn. I remember coming across this after traversing many, many links; I do remember
it was somewhere on the Microsoft tech notes area of their site.

I wonder how "verify after write" is defined? If the file is simply read
from the disk, without comparison to the original, to determine that it is
readable without disk errors, it would not rule out the kind of corruption
you describe. Because, from your description, the files are readable. It's
the contents that are corrupted. This rules out a disk write error. The
corruption is somewhere else in the process.

Good points.


Have you ruled out an application or driver problem? Audio drivers can
behave strangely in a multicore environment. Diginoise can be generated at
the moment of playback, it can even be repetitive, and it may occur simply
because the playback application is different, or being used in a different
way.

All machines are at the same revisions on all the common software; OS is at the same
rev (XP pro, SP3)...

Even drilling down deeper than that, we're talking about projects that had no issues
when last used. The only thing that's (mainly) changed is transfer (via USB) from
one drive to another, or perhaps a copy operation within the machine itself. (More
than once I've also discovered that a backup is bad as well, meaning that the
corruption took place way back in the chain.)

Bad hubs? Bad memory? Could well be, but like all things in troubleshooting, the
subtle intermittents will drive you a little crazy, particularly if they don't show
up until later.

Thanks again for all the input.

Frank
--

[Arny Krueger]

"Frank Stearns" wrote in
message
acquisition

Besides what seemed to be a robust file system, I also
miss the 300-400 days between boots on my SPARC stations.
Sigh.

Isolated XP systems running a reasonable but stable suite of software can go
100's of days between boots. Been there, done that.

[Richard Crowley]

"Arny Krueger" wrote ...
Isolated XP systems running a reasonable but stable suite of software can
go 100's of days between boots. Been there, done that.

Indeed we have hundreds of XP systems that run 364.9 days straight.
They turn them off once a year to blow out the cobwebs and dust bunnies.
Dunno why they can't keep the computer rooms any cleaner.

[Richard Crowley]

"Soundhaspriority" wrote ...
"Richard Crowley" wrote
posting from Botosani, Romania on tour.
I wish I was there!

How are you hooked up? "Cafe"? Are you using a VPN ?

Wi-Fi where we can find it. Currently at a 4-star hotel in Turgu
Mures, Romania but the Wi-Fi signal quality is "poor" in my room.
Some hotels have no Wi-Fi at all, and the newest ones have
*wired* (free) network, not wireless.

[Richard Crowley]

"Soundhaspriority" wrote ...
"Richard Crowley" wrote ...
"Soundhaspriority" wrote ...
How are you hooked up? "Cafe"? Are you using a VPN ?

Wi-Fi where we can find it. Currently at a 4-star hotel in Turgu
Mures, Romania but the Wi-Fi signal quality is "poor" in my room.
Some hotels have no Wi-Fi at all, and the newest ones have
*wired* (free) network, not wireless.

One of these
http://cgi.ebay.com/WiFi-Booster-Wir...3A1%7C294%3A50

with one of these:
http://cgi.ebay.com/USB-WiFi-Wireles...3A1%7C294%3A50

is the cure for that.

I like things with real SMB antenna connectors and visible antennas.
I can use nice, compact directional antennas like these...
www.wa5vjb.com

Although my WiMAX USB RF modem (which resembles a thumb
flash drive) seems to work just fine around the PDX market.

[Scott Dorsey]

Frank Stearns wrote:
Any particular "industrial strength" hw recommendations? (My first Sparc, for
example, WAS a disaster, but to Sun's credit, they did replace it. I'd like
to avoid that scenario again, though. The vendor was good about the issue but the
time is gone forever.)

Sun is still selling SPARC machines, but they have also ported Solaris to
The x86 platform, and they sell pretty bulletproof x86 machines. For
$2500 you can get a 1U rackmount box. Add anyone's cabinet full of disk
drives and you're good to go.

You could also buy a Dell rackmount machine and run Solaris or NetBSD
on it, but you wouldn't get the degree of support (although I do have to
say Dell's support for Linux isn't bad and Red Hat has a pretty damn
stable SMB service).

There are a bunch of commercial "Network Access Servers" out there, most
of which are secretly BSD or Linux boxes in disguise with a nifty web
interface so you don't have to see the underlying operating system. I
cannot say anything good or bad about any of these boxes in particular
but the idea is a good one and plenty of big sites are using them. If
you think an occasional glitch in an audio file is bad, just imagine what
an occasional glitch in a huge payroll database can do.
--scott

--
"C'est un Nagra. C'est suisse, et tres, tres precis."

[jason]

In article ,
says...
My experience with quad cores under XP is that they can behave oddly, but
never destructively.

I watched a semi-fascinating video by a senior MS architect
(Brad Mussinovich) about what's new/different in Win 7. He
said the single, largest change was for multi-core
machines.. and it was to get rid of the global dispatcher
lock that is present in all previous Win versions, including
Vista. Before the change, anything that would cause the
dispatcher to come into play was serialized on all the
processors. It could also lead to a "priority inversion"
situation, where a lower-priority task could actually
preempt one at higher priority. He said that under certain
circumstances, the serialization (and priority inversion)
could cause the machine to grind to a halt (well.. at
nanosecond speeds, a "halt" might be too brief to notice).
Lots of cores lead to lots of design headaches. Adding more
cores is not what Moore had in mind when he dreamt up his
"law." I worked on massively parallel machines and it's very
challenging to take advantage of all those processors...

(fwiw.. the latest Nvidia GPU has 240 cores... but they are
operating in a much more tightly-controlled environment than
the main CPU(s).)