resurrecting-open-source-projects / dcfldd Goto Github PK

Hi,

I am very interested in using dcdldd on macOS and Linux.
Homebrew distributes the "unrestricted version" yet yours is better feature wise.

Do you plan to distribute it over homebrew (Linux + Mac) ?

Regards,

David

The Debian Project has a CI system to test any package over the following architectures: amd64, armel, armhf, i386, ppc64el and s390x.

The following test fails on s390x:

dcfldd if=test.txt hash=md5,sha1,sha256,sha384,sha512 2>&1 | grep -C 20 bc0e4b74695142e0a0bdae87aea310d7078866cb

I attached the file test.txt to this issue. The grep shown above will test the SHA1 result. The other hashes are right. See below the result of the test over amd64 WITHOUT the grep:

Total (md5): 92994b0ce292a217e3e3bc31b639e565

Total (sha1): bc0e4b74695142e0a0bdae87aea310d7078866cb

Total (sha256): 02fd428a4671925e4ca61541b9fac648f4ccdccad65602bfd3256ba14e59489c

Total (sha384): 53b8374607a8258de4173265bfbfa6120093fd42090a92fd589cf2c6c16b4e421b5135777714976713f7949715720a83

Total (sha512): 969a39bf47b5f12d81121084f19cb5ae250e0c0ea5b7c6d82cf08131acde8e1955d137612c2f6b255e25b0e28f96f93586f90f06965cb8f719ed7fbdd95cc8d4

Now, over s390x:

Total (md5): 92994b0ce292a217e3e3bc31b639e565

Total (sha1): 68afccc873eb954866097c08d4ad9d9643735f2c

Total (sha256): 02fd428a4671925e4ca61541b9fac648f4ccdccad65602bfd3256ba14e59489c

Total (sha384): 53b8374607a8258de4173265bfbfa6120093fd42090a92fd589cf2c6c16b4e421b5135777714976713f7949715720a83

Total (sha512): 969a39bf47b5f12d81121084f19cb5ae250e0c0ea5b7c6d82cf08131acde8e1955d137612c2f6b255e25b0e28f96f93586f90f06965cb8f719ed7fbdd95cc8d4

Only SHA1 differ between both architectures. Executing the sha1sum command over the file on s390x will produce the right hash:

(sid_s390x-dchroot)eriberto@zelenka:~$ sha1sum dcfldd-1.7.1/debian/tests/test.txt 
bc0e4b74695142e0a0bdae87aea310d7078866cb  dcfldd-1.7.1/debian/tests/test.txt

test.txt.gz

Hi,
I tried dcfldd if=somthing.img of=/dev/sda of=/dev/sdb of=/dev/sdc

But it only writes to 2 of them and does not return any error. Please help.
My main goal is to write to multiple Rpi at the same time (around 10)

This Called Enhanced version of DD, but DD have speed info and this DCFLDD no have!
Whats reason for remove this default feature from orginal DD Tool?

Hi!

My impression is that the current Bash completion is not installed by the build system. I can manually copy the file at packaging stage of course but then every distro has to do that manually and before that there is no bash completion. Are there plans to make the build system install it for everyone?

Thanks and best, Sebastian

Bug from Debian[1].

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=886647

The original bug says:

Here is what I get when running dcfldd from one 50GB partition to another:

[0% of 28213188Mb] 400 blocks (400Mb) written. 117:33:11 remaining.

/proc/partitions has this for the partition in question:
8 2 56426376 sda2
i.e. it's 56 GiB and not 28TiB as dcfldd seems to think. It doesn't actually take 117 hours to complete just the usual 30 minutes.

The command line is:
dcfldd bs=1M if=/dev/sda2 of=/dev/sdf2 sizeprobe=if statusinterval=100

so I'm guessing that the sizeprobe is the problem.

I was able to reproduce it using the following command:

# dcfldd bs=1M if=/dev/sdb3 of=/home/test sizeprobe=if statusinterval=100
[0% of 14747648Mb] 300 blocks (300Mb) written. 68:16:29 remaining.

The /dev/sda3 has 28.1 GB, not 14.4 TB.

The next thread in the bug says:

Hi,
The cause of this bug is on
debian/patches/10_fix-probing-of-large-block-devices.patch so it is
Debian specific. The patch changes from using BLKGETSIZE to
BLKGETSIZE64 without correcting for the fact that BLKGETSIZE returns
the 512 byte sector size and BLKGETSIZE64 returns the byte size. This
is from /usr/include/linux/fs.h:

> #define BLKGETSIZE _IO(0x12,96) /* return device size /512 (long *arg) */
> #define BLKGETSIZE64 _IOR(0x12,114,size_t)      /* return device size  
> in bytes (u64 *arg) */

and this is the patch:

> --- dcfldd-1.3.4.1.orig/sizeprobe.c
> +++ dcfldd-1.3.4.1/sizeprobe.c
> @@ -63,9 +63,13 @@ static off_t midpoint(off_t a, off_t b,
>  static off_t get_dev_size(int fd, long blksize)
>  {
>      off_t num_sectors = 0;
> -
> -    if (ioctl(fd, BLKGETSIZE, &num_sectors))
> -        log_info("%s: ioctl call to BLKGETSIZE failed.\n", program_name);
> +
> +    /*
> +     * Use BLKGETSIZE64 unconditionally, since dcfldd.h #defines  
> _FILE_OFFSET_BITS 64
> +     * and off_t is guaranteed to be large enough to hold the result.
> +     */
> +    if (ioctl(fd, BLKGETSIZE64, &num_sectors))
> +        log_info("%s: ioctl call to BLKGETSIZE64 failed.\n", program_name);
>      else
>          return (num_sectors * 512);
>  }

Should be a trivial fix

However, this patch is not compatible with dcfldd now.

Eriberto

On Ubuntu 23.10 configure fails for a git checkout:

$ git clone https://github.com/resurrecting-open-source-projects/dcfldd.git
$ cd dcfldd/
$ ./autogen.sh 
configure.ac:48: warning: The macro `AC_HELP_STRING' is obsolete.
configure.ac:48: You should run autoupdate.
./lib/autoconf/general.m4:204: AC_HELP_STRING is expanded from...
configure.ac:48: the top level
configure.ac:33: installing './compile'
configure.ac:29: installing './config.guess'
configure.ac:29: installing './config.sub'
configure.ac:26: installing './install-sh'
configure.ac:26: installing './missing'
src/Makefile.am: installing './depcomp'
 
Done. You can use the 'clean' option to vanish the source code.
Example of use: $ ./autogen clean
$ ./configure 
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a race-free mkdir -p... /usr/bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
./configure: line 3248: PKG_PROG_PKG_CONFIG: command not found
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether gcc accepts -g... yes
checking for gcc option to enable C11 features... none needed
checking whether gcc understands -c and -o together... yes
checking whether make supports the include directive... yes (GNU style)
checking dependency style of gcc... gcc3
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking whether gcc needs -traditional... no
checking whether ln -s works... yes
checking whether make sets $(MAKE)... (cached) yes
checking for ranlib... ranlib
checking for egrep... (cached) /usr/bin/grep -E
checking for an ANSI C-conforming const... yes
checking for stdio.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for strings.h... yes
checking for sys/stat.h... yes
checking for sys/types.h... yes
checking for unistd.h... yes
checking whether byte ordering is bigendian... no
checking for off_t... yes
checking for size_t... yes
./configure: line 5434: syntax error near unexpected token `BASH_COMPLETION,'
./configure: line 5434: `    PKG_CHECK_MODULES(BASH_COMPLETION, bash-completion >= 2.0,'

From SF.net[1]:

[1] https://sourceforge.net/p/dcfldd/patches/1/

Status: open
Owner: nobody
Labels: None
Priority: 5
Updated: 2005-10-15
Created: 2005-10-15
Creator: Martin Scharrer
Private: No

dcfldd returns a floating point exception on my machine when started with 'statusinterval=0'.

This value should either be handled as invalid or should be aquivalent to 'status=off'. I wrote two small patches to add either one of this.

fix-exception.zip

Out of memory: Killed process 9737 (dcfldd-v1.9) total-vm:847432kB, anon-rss:844620kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:1696kB oom_score_adj:0

Prepare:

swapoff -a  # Just for a faster OOM
fallocate -l 10G test.dd  #create bigger file than your RAM

The bug:
dcfldd diffwr=on if=/dev/zero of=test.dd # OOM kill

Default option (without diffwr) isn't affected.

The problem:
When destination blocks are same (not written), memory blocks remain allocated.

Suggestion:
Use my "full_write" function, instead code in released v1.9

At first, thank you for commit.
And sorry, because I have no time to test @davidpolverari suggestion.

Dear @davidpolverari
Thank you for suggestion, but your code is BAD :(. And unfortunately your suggested code became part of the version, instead of mine code. Your code eat all memory, and trigger OOM kill in a real life. My code maybe not beautiful as your, but I tested weeks, and used on a real life work before I published.

I only noticed that the v1.9 release contains memory leak bug, because I installed it from a distribution instead of mine version. When I tired my routine job with "diffwr" option, I get OOM kill after dcfldd eat all memory. And this is caused by your proposed code.

But I tested with valgrind against my version vs released v1.9. And the suspicion turned out to be true. With diffwr=on option almost all malloc-ed block remain allocated. But only in your suggested code, not in my code.

I see only this leak in your (=v1.9 released) version:

valgrind -s --track-origins=yes --leak-check=full --show-leak-kinds=all ./dcfldd-v1.9 bs=1k count=1k diffwr=on if=/dev/sda of=dst.dd
...
==9430== 1,048,576 bytes in 1,024 blocks are definitely lost in loss record 17 of 17
==9430==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==9430==    by 0x113F23: full_write (full-write.c:56)
==9430==    by 0x11807C: outputlist_write (output.c:161)
==9430==    by 0x115017: dd_copy (copy.c:322)
==9430==    by 0x10C656: main (dcfldd.c:753)
...

The leak size almost same size as the (not) written data size. With GB (not) written, easy to eat all memory.

Remark:
My do..while(0) not nice I know, but this just a "simulated" try...throw..finally statement. The "break" is the "throw". :D Not nice, but my code always free the malloc-ed block. I don't know why, but your solution doesn't do that.

Maybe my code would have been better in the released version. Now the current release is buggy. :(

Originally posted by @szolnokit in #13 (comment)

There are new options to be put in file.

Hello,

i have read that someone is going to use following command:

sudo dcfldd if=/dev/sdb of=usb.img bs=3M hash=sha256 hashlog=sha256.txt; sha256sum usb.img

i read that by hashing may still not achieve 1:1 copy due to errors reading from source with bad sectors. Does it mean that dcfldd stop operation on read error? I have not found any "ddrescue" like switch to retry (-r) reading sectors that failed reading.

f="log/debug.log";time md5sum "$f";time sha256sum "$f"
shows:

real	0m0,008s
real	0m0,004s

so i assume that sha256 would be like 2x faster so i will use it just by defining hash=sha256 hashlog=sha256.txt; where hashlog parameter has no benefit of r€suming the interrupted process it is just to record it for convenience?

The README.md file doesn´t explain about the required build-time dependencies. There is a need to clarify those to ease the build and process by third-parties, especially contributors.

According to adulau/dcfldd#2, the usage of errlog= parameter causes dcfldd to segfault, as presented in the example on that repository (copied here verbatim):

# dcfldd if=/dev/zero of=test.raw bs=1 count=512 errlog=error.log
512 blocks (0Mb) written.
512+0 records in
Segmentation fault (core dumped)

The latest released tarball doesn't contain the configure script nor is ready for building without depending on autotools. The published release tarballs are expected to be the ones created with make dist. Users installing from source using the release tarballs shouldn't need autotools in order to build the program.

I am currently trying to clone my old HDD to an SSD, I would like to hash the whole thing. Unfortunately the hashlog file remains empty.

I used the following command:
dcfldd if=/dev/sdb of=/dev/sdc hash=sha512 hashlog=win10.hash hashwindow=512M bs=64K
the status is currently
357632 blocks (11176Mb) written.