QuickPar / par2 creating defective files

Kargor · 2009-09-02 19:08:13

Not even sure whether it's the correct program --- the par2 commandline utility that ships with Debian.

Code:

foo@gatekeeper:~> par2 --version
Not enough command line arguments.
par2cmdline version 0.4, Copyright (C) 2003 Peter Brian Clements.

It's one of the "lots of par2 blocks" cases; it has worked fine for me for a long time.

Code:

par2 create -s163840 -c13881 -u -n15 -v -v "foo.par2" ../6/0/0/X/*

Created files, but none of them contained any recovery blocks (as reported by QuickPar), and about 50% of them was "sparse" as well.

Code:

par2 create -s163840 -c13874 -f13881 -u -n15 -v -v "foo.par2" ../6/0/0/X/*

Was even worse --- it calculated and calculated and even gave the "wrote xxxxx bytes", but didn't create ANY file at all.

The files in question had these sizes: 1564689835, 781774364, 8095, 781813022, 8166, 781752473, 8297, 781822898, 1439

I hope names don't matter; the "par2" name as longer compared to what I usually get, but the created files had the correct names.

Last edited by Kargor (2009-09-02 19:25:57)

fat · 2009-09-02 20:40:21

Explicitly set the path for the created par2 files and try again

Kargor · 2009-09-03 20:06:27

I seriously doubt that will make a difference, but I'll run the dataset through it again to see what happens.

Currently trying to see if I can work around these bugs to get the intended result, rather than having to rearrange things...

... and the first attempt looks promising already: I simply reduced the number of recovery blocks (-c6475 -n7), and QuickPar now finds 925 blocks in each created file. So maybe I can create the intended result by just making multiple smaller runs, each starting at a different block number...
The created blocks acrtually seem to work as well (I don't know what QuickPar actually checks when it looks at the par2 files). Currently trying to restore a deleted file, but for some operations the name "QuickPar" is rather misleading.

It would be consistent with QuickPar, though --- it also likes to screw up for higher block counts. The question remains whether it's really the number of blocks, or the number of the blocks (which would make it fail again at the higher block numbers). Yes, I don't know how the algorithm works :-)

fat · 2009-09-03 22:39:25

Weird things hve happened in the past when no explicit paths were set hence the request to set them so we have a proper baseline to work.

Hopefully Peter will chime in here the math is beyond me

Yutaka Sawada · 2009-09-05 09:58:43

Hello, Kargor.
I am Yutaka Sawada. I don't know internal coding of QuickPar, but I feel your PC has not enough memory for your usage.

From your post;
block size = 163840 (160KB)
recovery block count = 13881

source files =
1564689835 (1492MB) -> 9551 blocks
781774364 (745MB) -> 4772 blocks
8095 (8KB) -> 1 blocks
781813022 (745MB) -> 4772 blocks
8166 (8KB) -> 1 blocks
781752473 (745MB) -> 4772 blocks
8297 (8KB) -> 1 blocks
781822898 (745MB) -> 4772 blocks
1439 (2KB) -> 1 blocks

source block count = 28643

If QuickPar keeps whole matrix on memory with 2-byte integer, the required amount is 28643 * 13881 * 2 = 795186966 (758MB). So, your PC must have at least 1GB memory, or 2GB memory may be good to work. If you have been succeed to create PAR2 file with so many blocks ago, you should check your PC's memory usage. If your PC has enough memory, but fail to create PAR2 file, there may be a bug in the client (par2cmdline/QuickPar). Try with different PAR2 client, like mine.

BTW, I think it is useless to create so many recovery blocks in PAR2. Even if you create 13881 recovery blocks by multiple smaller runs, you will not be able to recover those many blocks at once anyway. This is because PAR2 client uses Gauss/Jordan Elimination for decoding. Time to decode will be much longer as lost blocks become many. This is a problem of PAR2 spec, not QuickPar's fault. When you need to wait 1 week to finish calculation for repair, those are almost useless. My suggestion is that, use lower block count setting.

pcordes · 2009-11-29 04:41:04

Yutaka Sawada wrote:
Hello, Kargor.
I am Yutaka Sawada. I don't know internal coding of QuickPar, but I feel your PC has not enough memory for your usage.

He's not using QuickPar, he's using the GPLed par2cmdline. He didn't use -m, and the default is 16MB. I usually use -m1000 or something, since I have 4GB in my machine (running Ubuntu, so using the same version of par2 as Kargor). If you use -m less than the total size of the recovery set, then par2 will do the first chunk of each block and write that to disk, then the second chunk of each block and write that to disk (at offsets 1 chunk size higher.) If this process stopped half way through, you would be left with sparse .volxx+yy.par2 files

I haven't run into the problem you found. Kargor, what's your OS? i386 or amd64, and what version of Debian GNU/Linux? And did you already file a bug at bugs.debian.org? For anyone else reading, note that Debian's package of par2 includes patches that aren't in the sourceforge version of par2cmdline 0.4, e.g. the fix for -q quiet mode turning off incrementing a necessary counter, as well as not printing messages.

BTW, I think it is useless to create so many recovery blocks in PAR2. Even if you create 13881 recovery blocks by multiple smaller runs, you will not be able to recover those many blocks at once anyway. This is because PAR2 client uses Gauss/Jordan Elimination for decoding. Time to decode will be much longer as lost blocks become many. This is a problem of PAR2 spec, not QuickPar's fault. When you need to wait 1 week to finish calculation for repair, those are almost useless. My suggestion is that, use lower block count setting.

Depends on your use case. For archiving to DVD, where you don't expect to have to use the par2, having to wait a week (or a day, 5 years later when you notice the disk has faded, and recover it with your new faster PC) is not a bad price for having a higher chance of recovering at all. On optical disks, I think you definitely want to use as small a block size as you can get away with (but multiple of the DVD sector size of 2048, of course). Optical disks that start to decay because of age could easily lose scattered 32k all over the place. (32k for DVD outer-level ECC block size. CD ECC block size is much smaller, just 2k IIRC). Even scratches along a track won't go all the way around, so there will be
data data data SCRATCH data data
data data data SCRATCH data data
data data data SCRATCH data data
... repeat more depending on how with the scratch is.

My computer doesn't suck, so it's no problem to leave it on without crashing for weeks.

I have to admit, I didn't realize recovery times could be that bad, though. I usually use a 256k block size for backups onto 4.4GB DVDs. It takes a couple hours to calculate, and I didn't think repair would take a huge amount more than that.

#1 2009-09-02 19:08:13

par2 creating defective files

Code:

Code:

Code:

#2 2009-09-02 20:40:21

Re: par2 creating defective files

#3 2009-09-03 20:06:27

Re: par2 creating defective files

#4 2009-09-03 22:39:25

Re: par2 creating defective files

#5 2009-09-05 09:58:43

Re: par2 creating defective files

#6 2009-11-29 04:41:04

Re: par2 creating defective files

Yutaka Sawada wrote:

Board footer