Unbuffered custom copy/move

[admin]

Whoops. Now I had time to really look into it and saw why it did not work. Try the next beta... (and it will now also work in backgrounded processing).

[tedy]

Ok.
Meanwhile I was testing Teracopy custom handler to see if the "link" between the two programs is "pleasing" experience.
It doesn't seem to work . Maybe because Teracopy is a trial (free version). I thought it is because of the Server windows. Tried in a Win10 VM, the same. XY throws a dialog saying teracopy failed or something, and to try with Custom copy.

TeraCopy |%ProgramFiles%\TeraCopy\TeraCopy.exe
or
TeraCopy (RenameAll) |%ProgramFiles%\TeraCopy\TeraCopy.exe|/RenameAll
and other combinations.

[admin]

TeraCopy is supposed to work and AFAIK other users are using it with XY. Well, whatever caused your problem with it, you won't need TeraCopy for unbuffered copying in the next version.

[tedy]

Ok, thanks. I just mentioned it to not open a special bug report, if it's a problem with the last version or something. Maybe they use the registered teracopy version and the free version is not working with the said configuration, both on WIn2016 and Win10 for me. Fastcopy on the other hand, was working.

[jupe]

@tedy: Regarding Teracopy integration, I suggested using full paths to another user here, viewtopic.php?f=2&t=18501&p=156252#p156252

And he said that worked for him, I haven't tried myself though, but I am confident it has nothing to do with you using the free version.

[admin]

Whoops. Now I had time to really look into it and saw why it did not work. Try the next beta... (and it will now also work in backgrounded processing).

tedy, let me know if it works now.

[admin]

BTW, here is a little background read on buffered vs unbuffered copy:
https://blogs.technet.microsoft.com/ask ... py-issues/

I still would like to hear about any experiences with the new unbuffered copy option...

[admin]

tedy?

[tedy]

Hm?! Well, testing unbuffered I thought is easy enough for you to do it yourself? How come...
It can be (is) observed even on 1Gbps links although the negative effects on speed are much less pronounced.

No joy, still not working (6152 or whatever, or 4096 whatever it was..).
I've given up on this, forget it. I will just create my own program with all the logic I want specifically for me (maybe with configurability), see below.

Now I've invested some time and wrote my own simple copy program in C# that does unbuffered copy successfully. I'm not a Windows programmer (used to write small hobby things back in the days but not in recent 10 years) but in a matter of 1-2 hours I succeeded to do it in Visual Studio (old 2010 version). With overlapped/async IO, concurrently reading and writing. I used some articles on the internet. It turns out it wasn't as simple as just flipping a flag, maybe that's the culprit with Xy.

I would be able to wrap the logic in more pleasing UI and features if I know how to plug it in a custom External copy handler. I know Xy only supports fastcopy and teracopy. Fastcopy is a piece of sh* and it doesn't do unbuffered in my tests. And it had many kind of glitches, IIRC it failed to uninstall correctly and other nasty things.
Teracopy seems good, *it seems* does unbuffered but seems abandoned, was rumored to do unpleasant things to the system and I found some small glitches as well. In my prog I would be able to incorporate any logic I like for the process.
What does Xy pass to teracopy and how and what should I accept in my program to make it work? Maybe some API is used. Or if it's just some commandline parameters... A Custom External handler would be the best way I think.
This wasn't (isnt') my top priority (more are atuomatic backups with scripts over FTP which is fine over the fast network) so I may not be as active as before.

Edit: Nevermind. I figured it out with a test app - xy calls the teracopy exe with its command line parameters and I inspected them. I think I can pretend to be Teracopy and that's all. It seems Xy doesn't accept an entry if it has arbitrary Caption but it has to contain Teracopy. Anyway.
I tested the trial of Teracopy. All is good but it is way tooo slow to startup when I copy something, compared to native or even custom copy where when I press Ctrl+V it starts (and finishes for small copy jobs) in an instant!

[oblivion]

Teracopy seems good, *it seems* does unbuffered but seems abandoned, was rumored to do unpleasant things to the system and I found some small glitches as well.

I don't think it's abandoned: there's a beta (3.3) with a 2018 copyright date on it...

My only issue with Teracopy with XY (been using it for some time now) is sometimes invoking Move causes TC to throw "I can't find the file anymore" errors; I think maybe XY might be reacting to something early but I'm still trying to figure that one out.

[tedy]

The last official version is more than a year old. There is some beta from March, yes.
For me the main issue is TC starts slowly, 5-600ms. I copy things often and this lag I can't bear.
From what I tested, XY deletes the temporary file that holds the list of items to be copied only when exiting. Not tested with Move operations yet.
But thanks, I'll make my app to read once the temporary file and not relying it to be there all the time.
Fast loading and finishing of the copy/move handler is essential for me, so I'll try to finish my "project" with some flexibility, instead of TC.

[tedy]

Don.....
Hm, just was fiddling around again with CustomCopyFlags...
I had a glimpse at the next option in the ini file, named "CustomCopyNoBufferingFromBytes=0".
You missed this option in your previous posts.. or in the old changelog you quoted.
It seems the whole thing won't work when this is at its default value (0).
I tried and set it to 1 and voila, it worked - no memory/cache change when copying 10GB test file and the copy speed was steady from very start to finish (170MB/s) over the network.

So, it seems there is a bug in some condition where CustomCopyNoBufferingFromBytes=0 means "no unbuffered" or OFF instead of meaning all files with zero or more bytes filesize. Am I in the right track?

[admin]

The documentation of this tweak was lacking. I should have added these lines:

Code: Select all

      To invoke unbuffered copying for all non-empty files (= size 1 byte or 
      larger):
        CustomCopyNoBufferingFromBytes=1
      To disable unbuffered copying for all files:
        CustomCopyNoBufferingFromBytes=0

Note that CustomCopyNoBufferingFromBytes controls "no buffering" independently of CustomCopyFlags! If CustomCopyNoBufferingFromBytes is > 0 then you get unbuffered copying no matter whether/how you set CustomCopyFlags.

[tedy]

I'll reuse my old topic to share an overvation and to ask if this would be deemed normal.
I use unbuffered copy/move since then.
Now that I have two SSDs in my system. 250GB system drive (samsung 850 evo) and 1TB 860 evo which replaced an 860 evo 500GB.
Copying between the two with XY unbuffered custom copy results in the lesft side of the screenshot (~330MB/s constant speed and almost perfectly overlapping curves).
The right side is copying with Explorer (buffered) where it reads with max 500MB/s (with small interruption when it asked me there was an existing file in the destionation) and seesaw writing line. I'm not sure about the reasons for this writing line but I guess both "methods" result in about the same overall time of the operations. The right curves may not have been the same files or size, but they were similar nonetheless.

How would you comment the left curves and is it normal? Could it be improved to use most of the ssds' bandwidth? In tests (AS SSD) both drives look normal with ~530/490 MB/s read/write sequential.

(for full image - right click -> view image)

[admin]

XY does not control these operations on the byte level. It just tells Windows what to do via API calls. Only Windows could tell you how these curves come about.