XYplorer Beta Club

Hi.
Is there a way to introduce an option for unbuffered copy/move to Custom copy functionality (xycopy.exe)?

I have a 10Gbps link to a server from my desktop.
Up until now I was always using Windows Explorer to access the server through network shares.
Accessing the fast HDD on the server to upload files/archives was fast for relatively small files (several GB) - about 190-210MB/s.
The RAM files caches on both machines were big enough to contain some GB of the files so I didn't experience visible slowdowns.

The problem is with large files (10+GB). I conducted Many experiments last two days. Windows Explorer uses buffered copy/move and this proved to be a big problem - after the initial speed burst (500MB/s from an SSD) the transfer speed plummet to about 100MB/s to the very end. Average speed results to about 120-125MB/s.
When I use robocopy with the option /J (unbuffered copy), the speed is constant about 190MB/s on average. Using robocopy all the time is tedious and very unproductive.
I believe an option for unbuffered copy would be easy to implement.
Thanks.

Yes, I could add that. Good feature idea actually!

Question is: Make it a general setting ("Use no buffering" or so), or make it automatically set in from a certain threshold (files > 1 GB or so)?

Hi.
Actually I wasn't expecting to see enthusiasm for such a feature .
However you decide to implement it, I'd wish for the possibility to be able to make the threshold (if any) very low .
Most of the time when I copy files I don't want them buffered (cached), or they are already cached (small previously created archives).

As a first thought I can see it as one option, a digit: "0" could mean "do buffered as it has always been"; any digit > 0 (bytes, but could be conveniently shown as K,M,G units) would be "any file > this threshold would be unbuffered".
Or something like that.
In any case, a digit setting to customize the threshold would be best. I, presonally, would make all copy/move operations unbuffered. Windows has the bad habit to flush other and most likely important cache for working apps to make room for copied files that would almost never be needed, at least for me.
The degree to which buffering affects file copy/move operations depends largely on the amount of file cache Windows is utilizing for the operation, and it depends on the amount of free (available) ram (but not so much). That's my observations after many experiments.
Baesd on my tests the problem arises just when the target machine (the server) fills its defined cache size and starts immediately to flush (slowly) this cache (kind of parabola in task manager - Memory) - it's then when speed plummet drastically. For the server with 4GB free memory (available) this happens after about 2.5GB cache is filled. The opposite direction (towards my desktop with 12GB available) takes a bit more memory.
With robocopy /J, both machines never change their memory readings in task manager (the graph), even a 'bit' when I start a copy.

But Ok, after a second thought, it might be a simple list with 0 (no buffering at all), and 5-10 values from 100M to 10-50GB or something, up to "all buffered" where it would equal some very large value internally.
If it's easy, a field to enter a value for the threshold would be best. I see scenarios where the user might want small files to be buffered. For me personally, when I copy or move, it's for good, and most likely don't use the files afterwards.

P.S. After a third thought, a threshold option would be better though... - For files, say, 2GB the buffered copying would send the file in one 'breath' as is now - with 500MB/s over the wire and the target machine's RAM filecache would receive and contain the whole file in its cache in one burst and then it would take care to move data to HDD while for me (the desktop) the transfer will be finished fast. The filesize in this case depends on average free ram on the receiving machine one holds, and of course a factor is the desirable maximum cache I want a file_copy to occupy on the sending machine. So a customizable threshold would be best. I see it something like 1 or 2GB for my configurations.

Would that be a value per file, or the summary size of all files in the copy job? I assume the former.

Good question.
I can only speculate in my posts about such decisions and only say my opinions.

Well, maybe the latter.
My reasoning for this is: if I set the option to 1GB for example, I base this decision on the overall memory I would "sacrifice" for "copy cache operations". If I then copy 20 files x 0.9GB, it would totally negate this logic.
In this case I would just disable all buffering (0 threshold) to accomodate this logic.
I see scenarios (but maybe rare) where one would want to buffer smaller files even in multi-file copy-task.

MS docs are not helpful: "... unbuffered I/O, bypassing system I/O cache resources. Recommended for very large file transfers."

I read somewhere unbuffered copy involves calling CreateFile(Ex)() with the appropriate parameters for the unbuffered operation. Simple as that. At least that's what I have read. It works for Win Vista and up.

If I misunderstood your comment then I'm sorry.
"very large" is so vague when MS is involved in such matter..
I tried both methods (WE and robocopy) and for me "very large" where I see the benefit of unbuffered is anywhere around 10GB or more. The effect of buffering is seen on even smaller files but by a smaller margin.

Yes, it's totally easy to implement, just a flag.

Just saw that you can already activate COPY_FILE_NO_BUFFERING via a tweak: Play with it and let me know how it works.

Unfortunately it doesn't seem to work for me.
I set CustomCopyFlags=6152 (your two flags + 4096)
and it copies just as before - slowly. And I can see the target's task manager how the cache is climbing fast, then flushed and the speed stabilized around 100MB/s. Also this can be observed in Performance Monitor where Queue Length for the disk is almost constantly 2, while with unbuffered it is 0-1.
The same with 4096. The value is changed in a text editor while XY is not running.

Are you maybe using background processing? I just saw that I didn't implement it there.

Yes, by chance .

I'll try to turn off bg processing and edit this post.

Edit: Unfortunately no. The same story. I don't see how Bg processing is different in this case - the operation is handled by an external executable and main XY window is not blocked (one can work in it) either way.

Weird. If CustomCopyFlags=6152 then the flag is there and should work. Unless you have Windows XP:

Windows Server 2003 and Windows XP: This value is not supported.

And of course this has to be ticked:
Configuration | File Operations | File Operations | Custom Copy Operations | Use Custom Copy

Of course custom copy is checked, I can see the custom copy dialog with all the differences from the default windows dialog.
Windows server 2016. Robocopy command line tool with the /J option works.
When unbuffered is used you can see its effect easily in taskmanager where the memory graph doesn't change when you start a copy operation on big file. With buffered, the memory graph (on the target where you upload the file) climbs fast if the target disk is slower than the source. Source is SSD, target is single 3TB HDD with 180MB/s average (one third is full).
I don't know if only the flag to CreateFile is enough for this.

It's this flag: COPY_FILE_NO_BUFFERING

No hint on the web that there are any problems with it.

I don't know then.
I tested Teracopy the other day (in search of a GUI to perform my copy'es) and it also seems to support unbuffered because it also copies fast like robocopy. I tested Fastcopy as well but it seems to only copy buffered so I ditched it.
Maybe if XY could support arbitrary commandline exe in its custom copy handler ..
I'll make some more experiments with that flag tomorrow.