pixlsus / raw Goto Github PK

We should probably find a way to include the list of contributors that are found on this page on rawsamples.ch:

http://rawsamples.ch/index.php/en/donators

The text reads:

The following persons have given RAW-Files (unsorted order):

Ismael González, Martin Bloss, Reinhard Fetzer, Jens Duttke, Marco Rutz, Rainer Kriewald, Sergej Medvedev, Hans Schrotthofer, Marc Keune, Michi, Wolfram Soens, Ralf Geßner, Wolfgang Nickolay, Wolfgang Peisl, Andreas Pla., Oskar Teichmann, Martin Egger, Markus Schlieper, Peter Carsten, Uwe Mochel, Emil Eschenbach, Francis Willems, Ivaylo Iordanov, Gary Bainbridge, Ludger Jöster, Malcolm Barron, Andreas Norén, Michael Adams, David List, Urs Dünner, Dieter Bethke, Surinder Ram, Alexander Konopka, Udi Fuchs, Hans-Dieter Poppe, Bob Horton, Cyril Brunner, Niels Kristian Bech Jensen, Gerhard Hagen, Benjamin Adler, Jan Borgers, Frank Homann, Dirk Volkmann, Siegfried Henning, John Lightner, Gordon Goodsman, Yves Boyadjian, Paolo Massei, Elia Vecellio, Joerg Hoevel, Peter Gößweiner, Mike Newman, Thomas Lamprecht, Bernard Moschkon, Andrik Sieberichs, Dave Nicholson, Phil Harvey, Elia Vecellio, Suriya Matsuda, Matt Sephton, Tim McCormack, Zach Stuart, Horst Wittenburg, Dr. Falk Langhammer, Benjamin Derge, André Gärtner, Johannes Waschke, Anwar El Bizanti, Tim Beeck, Scott Picton, Adam Bryant, Heiko Kaufhold, Alexey Zilber, Andreas Isenegger, Radek Niec, Jean Glasser, Mats Karlsson, Willi Müller, Simon Schmitz, Till Grigat, Alexander Konopka, Alistair Jackson, Adrien Béraud, Aron Eisenpress, George Stiber, Robert Jackson, Phil Harvey, Sergej Medvedev, Marcel Cuculici, Chi Zhang, Fernando Prado, Prof. SAI GIRIDHAR KAMATH, Anders Torger, Herik Aiolfi, Marcel Wagner, Razil Shaikh, Haiyan Qu, James Wyper and many others.

Thanks a lot to all of them for their support and help
Jakob Rohrbach.

https://raw.pixls.us/getfile.php/820/exif/Sony_A6500_12Bit_Silent_Mode.ARW.exif.txt
https://raw.pixls.us/getfile.php/819/exif/Sony_A6500_14Bit_Normal_Mode.ARW.exif.txt

this needs to be listed in Of the following brands we like to have:

http://www2nd.sigma-photo.co.jp/downloads/manual/DP2Merrill_Manual_en.pdf, page 63
even raw files will have different image size

but that's not mentioned at raw.pixls.us

Somewhat related to #30 is the fact that the different users of the sample set have different needs. As I am working on RawSpeed I am interested in a set of files that covers as much (distinct) camera models as possible, while being minimal at the same time. My current regression test with ASAN enabled takes a few minutes on the complete set, which is not really sexy already. What I do not care about is different ISO settings for the same model or even different files coming the very same hardware just with a different branding. Different ISO samples on the other hand are very useful to come up with proper WhitePoint numbers.

Bottom line: Ideally I'd like to be able to somehow mark files as 'not interesting' for rawspeed regression tests and then later download/sync only those that are (see #25). The question remains on how I would
find out which are interesting and which are not.

A good first guess would be to have only one from each "make / model / mode".

I could envision some kind of automatism using code coverage analysis and then adding files and one that does not increase the coverage any more gets to be ignored in the future.

Not sure if all that would be worth it, it would be just nice if I had it (right now) ;).

For local, non-automatic use.
NOT for auto-download from CI.

How do i download all the samples?

It is probably a good idea to have a Captcha on file upload so we don't get spammed with a zillion false uploads.

just like in user ui

since we decided that cc0 is better suited for us

currently unter raw there is only

pic_1-1.RWL
9d83454a250dc18ca59b9b842d834a6b77840b14(11.97MB)

without md5, sha1, ...

next to Pentax: Both PEF and DNG.
there should be a line: Canon CHDK: both the old-style CRW, and new DNG
or something like that

For leica, there is no differentiation between compressed and uncompressed raw files.
https://raw.pixls.us/getfile.php?type=exif&id=752
https://raw.pixls.us/getfile.php?type=exif&id=753

diff points at

-Exif.SubImage1.BitsPerSample                  8
+Exif.SubImage1.BitsPerSample                  16

Now that some images are being replaced with image licensed CC0, i'd prefer to see that i actually deleted the right image, not the new CC0 variant...

@patdavid

[11:45:10] <schenlap> for "if the sample we have is not under CC0" we should add the necessary symbol from the table to explain it better.
[11:45:22] <schenlap> .. on raw.pixls.us

Since there is a mail already when a new sample being uploaded,
i think there should be similar mail when a sample being deleted, with the info about
who did delete it (name+mail)

so many cameras have this setting. it should be always present, not in mode.
#34

For GITUP-GIT2-4:3.RAW, i see 4b0118aaffe8c4f6e8edd8b163e450d5, while sha1sum is 4b0118aaffe8c4f6e8edd8b163e450d516b99eee

Can getfile.php output some header so that wget works?

$ wget https://raw.pixls.us/getfile.php?type=archive
--2017-01-02 22:42:54--  https://raw.pixls.us/getfile.php?type=archive
Resolving raw.pixls.us (raw.pixls.us)... 145.24.145.156
Connecting to raw.pixls.us (raw.pixls.us)|145.24.145.156|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6841295589 (6.4G) [application/zip]
Saving to: ‘getfile.php?type=archive’

getfile.php?type=archive                                            100%[==================================================================================================================================================================>]   6.37G  8.65MB/s    in 13m 24s 

2017-01-02 22:56:18 (8.12 MB/s) - ‘getfile.php?type=archive’ saved [6841295589/6841295589]

I'd like that to create some better filename, like raw-pixls-us-archive-$YYYYMMDD.zip
Same goes for https://raw.pixls.us/getfile.php?type=raw&id=129, it should create RAW_CANON_EOS_7D-sraw.CR2

@patdavid it may be wip, but branding on rpu seems too large, it should be half of that, and maybe not at the top, but in the background, or to the right/left of the text

https://discuss.pixls.us/t/new-year-new-raw-samples-website/3057/7?u=lebedevri

The user has to agree that "The file is manually copied from card/camera, without using any software like Nikon Transfer."

But the user does not have to agree that they didn't modify the file after copying it from the camera. I'm surprised. You will receive files that have been modified on the PC to include GPS data, date-time shifts, and other metadata changes, as rawsamples.ch did.

https://discuss.pixls.us/t/new-year-new-raw-samples-website/3057/6?u=lebedevri

Looking at

e.g. https://raw.pixls.us/getfile.php?type=exif&id=837
Exif.SubImage2.BitsPerSample 14

So i have tried to place shields on http://www.darktable.org/resources/camera-support/
And they are getting blocked by Privacy Badger.

E.g. https://raw.pixls.us/getfile.php?type=raw&id=741 and https://raw.pixls.us/getfile.php?type=raw&id=742

If

Exif.CanonCs.SRAWQuality                      n/a

then

Exif.CanonCs.Quality                          RAW

should be used.

See Canon EOS 70D samples.

http://helpguide.sony.net/ilc/1450/v1/en/print.pdf
says that there is

Compressed:
Records images in compressed RAW format.
Uncompressed:
Records images in uncompressed RAW format. When [Uncompressed] is selected for [
RAW File Type], the file size of the image will be larger than if it were recorded in
compressed RAW format.

so Of the following brands we like to have: should read Sony: all the combinations of compression and bitness settings, that is: 12bit-compressed 12bit-uncompressed 14bit-compressed 14bit-uncompressed

and if possible the compression should be detected, i'm not sure yet which fields that is

In https://github.com/darktable-org/rawspeed/blob/develop/README.md
There is a string: is currently tested on more than 500 unique cameras.
I would prefer to replace hard coded constant with such auto icon.
Something like the ones on https://shields.io/ although not necessarily using that service.
Also, maybe an icon with total number of samples

And results in local move /tmp/phpEvlZ4w being displayed
Also, resulting file does not denote license

See GITUP-GIT2-16:9.dng

We do not specify that we want more than one sample per camera.
We should say which samples we want, that is:

• Panasonic: all the crops (1:1, 4:3, 3:2, 16:9)
• Nikon: all the combinations of compression and bitness settings, that is 12bit-compressed, 12bit-uncompressed, 12bit-lossless-compressed, 14bit-compressed, 14bit-uncompressed, 14bit-lossless-compressed,
• Canon: RAW, mRAW, sRAW (FIXME: there are different gradations for m/s-RAWs, do we want just one mRAW and one sRAW ?)
• ???

https://raw.pixls.us/edit-admin.php?id=952
https://raw.pixls.us/edit-admin.php?id=953
https://raw.pixls.us/edit-admin.php?id=954
https://raw.pixls.us/edit-admin.php?id=955
they all compute as 3:2
is that actually true though?

https://raw.pixls.us/getfile.php?type=sha1sum&id=129 asks me where to save it.
surely the checksum is just a text string which can be displayed in browser

If we open the source code for https://github.com/darktable-org/rawspeed/blob/develop/README.md
we can see:

<img src="https://camo.githubusercontent.com/<blurb1>/<blurb2>" alt="raw.pixls.us - the free raw sample archive" data-canonical-src="https://raw.pixls.us/button-cameras.php" style="max-width:100%;">

So it caches them. Perhaps some header can be outputted to decrease the cache time?

the blurb before the actual content (the table) is getting too long.
it it tedious to scroll through it.
i would like to have a link at the top, that would scroll down to the table

We do accept cc0 samples if there is non-cc0 sample.
But it is not really stated anywhere.
I guess

If you can provide a raw file from a camera we're currently missing, or if you can provide a more useful photo from a camera model we already support

should be

If you can provide a raw file from a camera we're currently missing, if the sample we have is not under CC0, or if you can provide a more useful photo from a camera model we already support

The "Leica D-Lux (typ 109)" support mode 1:1 3:2 4:3 and 16:9 according to the user manual.
https://redmine.darktable.org/issues/11288 has the samples

I'm just looking at the overall table layout and I was wondering if we really needed all of the columns visible all of the time?

For instance, the checksum column. Is this something that we could make available as a text link similar to how EXIF data currently works? That column alone could compress the overall table width nicely, I think.

An option in it's place would be a link to the checksum, showing it on hover, etc...

In the meantime I've made the font smaller, monospace, and allowed the break-word to break-all.

So there is one more issue, e.g. nikon can produce raw files with different crop/MP count
https://raw.pixls.us/getfile.php?type=exif&id=755
https://raw.pixls.us/getfile.php?type=exif&id=211
https://raw.pixls.us/getfile.php?type=exif&id=193
https://raw.pixls.us/getfile.php?type=exif&id=208
As far as i can tell, only Exif.SubImage2.ImageWidth and Exif.SubImage2.ImageLength are different.

Perhaps there should be a fields that contain values of these fields, and they should also be taken into account when checking for sample duplication

While showing the thumbnail in the list of will be way too intrusive, in the edit page there is a lot of unused page space

#35, #33

Is there any better way than just downloading the whole archive?

Add some kind of upload timestamp to the table, so I can sort by that and see what is new.