The nightmare of long copy jobs: Solutions for the failures?

[Mesh]

A lot of work over many years of experience has taught me that large copy jobs have an alarming tendency to fail, and need to be split up into several smaller jobs in order to succeed.

This can occur during a local copy, from one drive to another - but it is especially prominent during network transfers.

Sometimes, you simply get an error message. Other times, you get a crash. Just the other day, I ran a network copy job, transferring about 650 GB in a single run. A couple of hours into the copy, the machine I was copying from got a BSOD. Obviously, you can't trust a fractional copy job, so I had to delete everything that made it over and start all over again.

The next day, during a similarly large network copy job, it was the *target* machine that got a BSOD.

But transferring everything over in bite size pieces (say, 50-80 GB) worked just fine.


I'm not sure if the weakness is in the O/S, or in NIC drivers, but I've encountered this on countless machines with varying hardware and O/S's from Windows 3.11 to Windows 7, as well as Macs.


I haven't yet tested whether or not splitting it up into bite sized pieces via XY's background jobs addresses the issue. But the truth is that I feel more comfortable with the 10-30 second delay that occurs when I set up the next bite sized piece. I figure that small delay gives the machine(s) a chance to flush whatever cache or queue is getting choked up.


Is there any way to add a delay in between queued jobs in XY?

Alternately, if anyone has any knowledge about this issue, I'd be glad to hear any insights you may have.

[admin]

Is there any way to add a delay in between queued jobs in XY?

Currently not, but would be doable. I just don't know if this helps. I personally never experienced a failed copy so I cannot really add to the debate.

[Mesh]

Is there any way to add a delay in between queued jobs in XY?

Currently not, but would be doable. I just don't know if this helps. I personally never experienced a failed copy so I cannot really add to the debate.

I understand. Most people don't usually have massive transfer jobs, so they don't really come across that issue.

[admin]

Is there any way to add a delay in between queued jobs in XY?

Currently not, but would be doable. I just don't know if this helps. I personally never experienced a failed copy so I cannot really add to the debate.

I understand. Most people don't usually have massive transfer jobs, so they don't really come across that issue.

Do you have any evidence that such a delay would help?

[Mesh]

Do you have any evidence that such a delay would help?

Well, what helps is splitting up a large job into several smaller ones. The problem is that that requires personal attention for each step. The ability to queue jobs the way XY can is fantastic - but if my guess is right that both sending and receiving systems need a few seconds to purge their respective queues/caches, being able to manually add a delay between background jobs would allow them to be automated so that a user didn't have to stay to handle every segment.

It occurs to me that rather than manually entering a delay each time one is desired, a better way to do it would be to simply have a single option field to enter the number of seconds that XY waits between all background jobs. A setting of zero would match XY's current behavior.

If you decide to do this, that might not be a bad way to implement it.

[admin]

I'll add a tweak.

[Mesh]

I'll add a tweak.

Thanks! Much appreciated!

[admin]

Look for INI key MsecDelayNextQueuedJob in the next version (run+exit XY to have the key created in the INI). I won't doc it in the change log.

[Mesh]

I'll keep an eye out. Thanks again!

[admin]

I'll keep an eye out. Thanks again!

It's already up in today's beta version.

[Mesh]

I'll keep an eye out. Thanks again!

It's already up in today's beta version.

I just grabbed it. I'll install it as soon as I get a chance.

[Mesh]

Don, I'm a touched confused about the comment - "give time (in msecs) to other threads". That seems to imply something completely different. Am I just reading it wrong?

[admin]

Don, I'm a touched confused about the comment - "give time (in msecs) to other threads". That seems to imply something completely different. Am I just reading it wrong?

You said:
"I figure that small delay gives the machine(s) a chance to flush whatever cache or queue is getting choked up."

Maybe it's my English but I meant say that XY steps back for a certain amount of time and lets all other threads work out any pending jobs.

[Mesh]

Ah, now I see what you were saying. It was just a bit confusing to me, and I was a part of this thread - users who haven't seen this thread won't understand what that tweak does at all. It might be worthwhile to label it more along the lines of "Time (in ms) between consecutive background jobs".

Sorry about the confusion!

[admin]

Ah, now I see what you were saying. It was just a bit confusing to me, and I was a part of this thread - users who haven't seen this thread won't understand what that tweak does at all. It might be worthwhile to label it more along the lines of "Time (in ms) between consecutive background jobs".

Sorry about the confusion!

OK, I always take the advice of a first language speaker.