fcopy
CONTENT

Overview

fcopy - utility to copy files. Before copying the file fcopy finds its total size and requests OS to allocate free space for a new copy in advance. After space is allocated fcopy starts multiple worker threads which read data from the source file in blocks of certain size and write them to the destination file.

At that moment fcopy does not support data integrity verification after the files were copied. Please use md5sum/crcsum if needed. fcopy relies on Java IO and if it does not throw an exception then fcopy considers data as being written to the disk without any corruptions.

Download

You can download fcopy from here

Requirements

Java 11

Usage

fcopy [ OPTION ]... SRC DST

Where:

Options:


-bs BLOCK_SIZE - when fcopy reads and writes data it does it in blocks of given size in bytes (default is 256K as the most popular among SSD drives)


-t NUMBER_OF_THREADS - number of worker threads which will copy the data in parallel (default is number of cores in the system)


-c QUEUE_CAPACITY - size of internal queue from which worker threads pick up the blocks they need to copy (default is NUMBER_OF_THREADS^2)


-s - enables silent mode

Examples

Copy ~/test file to /tmp directory:

% fcopy ~/test /tmp Configuration: Threads: 11 Queue size: 121 Block size: 256000 Copying /home/ubuntu/test to /tmp/test

Measurements

Below are the measurements between fcopy, rsync and cp.

Single file

Test files:

fcopy

Default:

% time -p fcopy ubuntu-18.04.1-desktop-amd64.iso /tmp/l Configuration: Threads: 11 Queue size: 121 Block size: 256000 Copying ubuntu-18.04.1-desktop-amd64.iso to /tmp/l real 4.24 user 2.07 sys 6.81

With 1MB block size:

% time -p fcopy -bs 1000000 ubuntu-18.04.1-desktop-amd64.iso /tmp/l Configuration: Threads: 11 Queue size: 121 Block size: 1000000 Copying ubuntu-18.04.1-desktop-amd64.iso to /tmp/l real 2.53 user 1.58 sys 4.94

With 30 threads and 2048000 block size:

% time -p fcopy -bs 2048000 -t 30 ubuntu-18.04.1-desktop-amd64.iso /tmp/l Configuration: Threads: 30 Queue size: 121 Block size: 2048000 Copying ubuntu-18.04.1-desktop-amd64.iso to /tmp/l real 1.89 user 1.11 sys 3.42

rsync

% time -p rsync -W ubuntu-18.04.1-desktop-amd64.iso /tmp/l real 4.35 user 5.50 sys 1.85

cp

% time -p cp ubuntu-18.04.1-desktop-amd64.iso /tmp/l real 1.43 user 0.02 sys 1.12

Multiple files

Test folder:

fcopy

Default;

% time -p fcopy ubuntu /tmp/l Configuration: Threads: 11 Queue size: 121 Block size: 256000 Copying ubuntu/lubuntu-18.04-alternate-amd64.iso to /tmp/l/lubuntu-18.04-alternate-amd64.iso Copying ubuntu/lubuntu-19.04-desktop-amd64.iso to /tmp/l/lubuntu-19.04-desktop-amd64.iso Copying ubuntu/ubuntu-18.04.1-desktop-amd64.iso to /tmp/l/ubuntu-18.04.1-desktop-amd64.iso Copying ubuntu/ubuntu-18.04.1-server-amd64.iso to /tmp/l/ubuntu-18.04.1-server-amd64.iso Copying ubuntu/ubuntu-19.10-server-amd64.iso to /tmp/l/ubuntu-19.10-server-amd64.iso real 21.61 user 5.40 sys 31.32

With 2048000 block size:

% time -p fcopy -bs 2048000 ubuntu /tmp/l Configuration: Threads: 11 Queue size: 121 Block size: 2048000 Copying ubuntu/lubuntu-18.04-alternate-amd64.iso to /tmp/l/lubuntu-18.04-alternate-amd64.iso Copying ubuntu/lubuntu-19.04-desktop-amd64.iso to /tmp/l/lubuntu-19.04-desktop-amd64.iso Copying ubuntu/ubuntu-18.04.1-desktop-amd64.iso to /tmp/l/ubuntu-18.04.1-desktop-amd64.iso Copying ubuntu/ubuntu-18.04.1-server-amd64.iso to /tmp/l/ubuntu-18.04.1-server-amd64.iso Copying ubuntu/ubuntu-19.10-server-amd64.iso to /tmp/l/ubuntu-19.10-server-amd64.iso real 8.08 user 4.25 sys 10.87

With 4096000 block size:

% time -p fcopy -bs 4096000 ubuntu /tmp/l Configuration: Threads: 11 Queue size: 121 Block size: 4096000 Copying ubuntu/lubuntu-18.04-alternate-amd64.iso to /tmp/l/lubuntu-18.04-alternate-amd64.iso Copying ubuntu/lubuntu-19.04-desktop-amd64.iso to /tmp/l/lubuntu-19.04-desktop-amd64.iso Copying ubuntu/ubuntu-18.04.1-desktop-amd64.iso to /tmp/l/ubuntu-18.04.1-desktop-amd64.iso Copying ubuntu/ubuntu-18.04.1-server-amd64.iso to /tmp/l/ubuntu-18.04.1-server-amd64.iso Copying ubuntu/ubuntu-19.10-server-amd64.iso to /tmp/l/ubuntu-19.10-server-amd64.iso real 7.41 user 4.07 sys 9.34

rsync

% time -p rsync -rW ubuntu /tmp/l real 14.31 user 17.36 sys 5.72 % time -p rsync -r ubuntu /tmp/l real 13.93 user 17.62 sys 5.02

cp

% time -p cp -rf ubuntu /tmp/l real 5.87 user 0.04 sys 3.55