Multithreaded File Copy Is Far Slower Than A Single Thread On A Multicore Cpu

I am trying to write a multithreaded program in Python to accelerate the copying of (under 1000) .csv files. The multithreaded code runs even slower than the sequential approach. I

Solution 1:

Of course it's slower. The hard drives are having to seek between the files constantly. Your belief that multi-threading would make this task faster is completely unjustified. The limiting speed is how fast you can read data from or write data to the disk, and every seek from one file to another is a loss of time that could have been spent transferring data.

Solution 2:

I think I can verify that it is a disk I/O situation. I did a similar test on my machine, copying from an extremely fast network server back onto itself and I saw nearly a 1:1 speed increase just using your code above (4 threads). My test was copying 4137 files totaling 16.5G:

Sequential copy was 572.033 seconds.
Threaded (4) copy was 180.093 seconds.
Threaded (10) copy was 110.155
Threaded (20) copy was 86.745
Threaded (40) copy was 87.761

As you can see there is a bit of a "falloff" as you get into higher and higher thread counts, but at 4 threads I had a huge speed increase. I'm on a VERY fast computer with a very fast network connection, so I think I can safely assume that you are hitting an I/O limit.

That said, check out the resonse I got here: Python multiprocess/multithreading to speed up file copying. I haven't had a chance to try this code out yet but it is possible that gevent could be faster.

• Spencer

Solution 3:

I assume this is more a I/O bound task, multithread should help the operation speed, anything wrong with my approach?!

Yes.

1. Too many punctuation marks. Just one. "?" is appropriate.

2. Your assumption is wrong. Multithreaded helps CPU bound (sometimes). It can never help I/O bound. Never.

All threads in a process must wait while one thread does I/O.

or coroutine to do the job?!

No.

If you want to do a lot of I/O, you need a lot of processes.

If you're copying 1000 files, you need many, many processes. Each process copies some of the files.

Solution 4:

as an aside I just wanted to add that the above code is slightly wrong. You should call fileQueue.task_done() AFTER shutil.copy(fileName, destPath) .. otherwise the last files will not be copied :)

Solution 5:

There exist cpu bounded applications and i/o bounded applications, normally you can get almost linear benefit from multithread app when its sequential version is cpu bounded. But when you are i/o bounded you are going to gain nothing, many operating systems can show you the "busy time percentage" of you CPU and the "disk busy time percentage" too, that way you can know which is your case.

BUT, due to normally the sequential code is not async, you end fetching one file, and then waiting for that file copy, then next file. That way you avoid the Operating system have the list of files and prioritize the read requests based on surface disk location.

Conclusion: if you look for maximum performance go single-thread but using Async APIs to allow the OS schedule the read requests better.