RAW output information

[bablokb]

So those are the CCMs and AWB gains that the internal algorithms have chosen. Half the point of capturing raws is that you can tinker with them, as otherwise you are just going to recreate the same image as the GPU (although you could tweak values in other parts of the processing chain).

The other half of the point is that you have a good starting value for optimizing colors (unless the camera is completely off). Every decent raw converter offers the choice to set AWB values to the values the camera has choosen - if they are available.

Independently of this argument, you just need a color-matrix. Raspiraw just uses an identity-matrix which is way off, while dcraw uses a hard-coded default. Not too bad for average light situations, but not every situation has average light.

[jbeale]

I have updated http://bealecorner.org/best/RPi/ to include a RAW sample from raspistill using November 2014-era firmware.

Supposedly this color matrix is reasonable: http://www.raspberrypi.org/forums/viewt ... 50#p511476

[tchiwam]

This seems to be the most neutral and fastest back to back raw I can have

raspistill -v -set -n -bm --raw -th none -awb off -awbg 1.0,1.0 -drc off -tl 0 --quality 100 -t 43200000 -ss 6000000 -ISO 800 -l latest.jpg -o `date +%Y%m%d-%H%M%S`-12H-ISO800-6000ms-%05d.jpg

2014-11-29 01:13:51.133117356 +0000 20141129-003315-12H-ISO800-6000ms-00193.jpg
2014-11-29 01:14:03.732839072 +0000 20141129-003315-12H-ISO800-6000ms-00194.jpg
2014-11-29 01:14:16.282561936 +0000 20141129-003315-12H-ISO800-6000ms-00195.jpg
2014-11-29 01:14:28.782285936 +0000 20141129-003315-12H-ISO800-6000ms-00196.jpg
2014-11-29 01:14:41.322009108 +0000 20141129-003315-12H-ISO800-6000ms-00197.jpg
2014-11-29 01:14:53.811733416 +0000 20141129-003315-12H-ISO800-6000ms-00198.jpg
2014-11-29 01:15:06.391455789 +0000 20141129-003315-12H-ISO800-6000ms-00199.jpg

Then I'll use exiftool to get the creation time, would be nice to add the exif timestamp of t0 and scan clock

[bablokb]

I have updated http://bealecorner.org/best/RPi/ to include a RAW sample from raspistill using November 2014-era firmware.

Supposedly this color matrix is reasonable: http://www.raspberrypi.org/forums/viewt ... 50#p511476

Thanks a lot, I will check the new image this afternoon. The color matrix from the quote is actually the hard-coded default from dcraw and it is indeed reasonable (at least for outdoor shots).

[jbeale]

I got dcraw.c to work on new & old RAW images without needing to use exiftool. See also: http://www.raspberrypi.org/forums/viewt ... 59#p647559

[bablokb]

I got dcraw.c to work on new & old RAW images without needing to use exiftool. See also: http://www.raspberrypi.org/forums/viewt ... 59#p647559

I have updated the raspi_dng code (see https://github.com/bablokb/raspiraw). This version also accepts a color-matrix as an optional third parameter. The new wrapper script raspi_dng.sh extracts the matrix from the EXIF-data and then calls raspi_dng for the actual conversion using this matrix.

I still have to gain some insights into this matrix though. Scaling seems technically not necessary, but scaling the values down to around 1 gives better results (at least from first experiments and using rawtherapee to process the dng-file).

[tchiwam]

Are the raw values going to be changed or is this matrix is stored so that the display program knows what to do with the data ?

[jbeale]

I have updated the raspi_dng code (see https://github.com/bablokb/raspiraw). This version also accepts a color-matrix as an optional third parameter. The new wrapper script raspi_dng.sh extracts the matrix from the EXIF-data and then calls raspi_dng for the actual conversion using this matrix.

I still have to gain some insights into this matrix though. Scaling seems technically not necessary, but scaling the values down to around 1 gives better results (at least from first experiments and using rawtherapee to process the dng-file).

Good work. I'm not sure I understand color matrix operations, but one difference with scaling is if the values are large, color values might get clipped during intermediate computations or at output. If they are too small, might loose precision due to roundoff. I don't know if those are done in floating-point and then rescaled, or (faster but more dangerous) in fixed-point format.

[jamesh]

Internally, IIRC, the ISP does calculations in 16p16 fixed point.

[6by9]

Internally, IIRC, the ISP does calculations in 16p16 fixed point.

My recollection is the same. Although they are generally saturating arithmetic operations so you don't get odd wrap arounds in values. Floating point is just too expensive on power, silicon area, and generally time to be considered viable (the ISP in BCM2835 is processing a pixel per clock cycle. Later versions could do up to 4 pixels/clock).

[bablokb]

Are the raw values going to be changed or is this matrix is stored so that the display program knows what to do with the data ?

Raspiraw just stores the matrix in a TIFF-header field. The converter/display-program has to take care of the data.

[tchiwam]

-awbg 1.0,1.0

If this analog gain on the R and B channel or digital multiplier ? And does it in anyway affect the raw data ?

[6by9]

-awbg 1.0,1.0

If this analog gain on the R and B channel or digital multiplier ?

Digital multiplier. I haven't come across any sensor where you can set different gains for the different channels.

And does it in anyway affect the raw data ?

Nope. Those gains are applied in the BCM2835 ISP.

[ethanol100]

Just a small remark: If you set values which do not make much sense(i.e., 0.001,0.001). The auto exposure algorithm can than change the analog gain, which would change the raw data.

[aphextwin]

hi there,

I have problems with the raspi_dng:
I'm getting compilation-errors when trying to compile on rip and on a ubuntu 64 bit system.
Can somebody please upload a compiled binary of the raspi_dng ??


thanks a lot!

[jbeale]

I have problems with the raspi_dng:
I'm getting compilation-errors when trying to compile on rip and on a ubuntu 64 bit system.

If you specify what specific errors you see, and which source you are using, eg. is it the newer
https://github.com/bablokb/raspiraw or the code from last year https://github.com/illes/raspiraw
...then there is a better chance of solving the issue.

[aphextwin]

sorry, you're absolutely right.

I tried to compile the newer version on my rpi b+.
The Illes-version compiled like a charm but the newer did not work at the very end. It seemed a tiff-library was missing.
I'll get the compile-log later, I don't have it right now.

What I try to do is using Imatest, of which I own a license, to measure MTF of different lenses.

I kind of solved my issue right now by writing the new rgb-values into the old Illes-code, which works fine by now.

Anyways I wanted to thank you all for your posts in this thread which are a great contribution!!

[bablokb]

I have problems with the raspi_dng:
I'm getting compilation-errors when trying to compile on rip and on a ubuntu 64 bit system.

As stated in the readme, compiliation on rip is currently broken. I didn't have access to my pi for a couple of days, I will fix that soon. Setting LIBSUFFIX to empty probably should do the trick. This is just a hack, I have to figure out something better (automake or cmake just seems overkill for such a small program).

Can somebody please upload a compiled binary of the raspi_dng ??

I added a subdirectory 'bin' to my repository https://github.com/bablokb/raspiraw with a x64-binary of raspi_dng. This might or might not run on your system.

[aphextwin]

Thanks, I just the old one in which I altered the rgb-values. Take your time.

[bablokb]

Thanks, I just the old one in which I altered the rgb-values. Take your time.

I fixed the makefile, it works again on the RPI. I also added the binary I just compiled to the bin directory of https://github.com/bablokb/raspiraw.

[aphextwin]

Thanks, I'll try to compile it when I'm at the office and give a feedback.

[tchiwam]

Just in local do this:
$ cd local
$ ln -s lib lib64

gcc: error: local/lib64/libtiff.a: No such file or directory
Makefile:32: recipe for target 'raspi_dng' failed
make: *** [raspi_dng] Error 1

[aphextwin]

Hi,

the new raspi_dng-binary works on my rpi, thanks for that.

As a student I was able to get my hands on a Imatest-Master-License and at my lab we've also got a spectrophotometer with which I can measure Lab and XYZ-values.
What I'd like to know is how can I refine the rgb-matrix with Imatest as I am new to this program? Of course I will share my results so we all get better images.

[bablokb]

[quote="aphextwin"
What I'd like to know is how can I refine the rgb-matrix with Imatest as I am new to this program? Of course I will share my results so we all get better images.[/quote]

No idea about imatest, but in http://ninedegreesbelow.com/photography ... ofile.html you will find a discussion on how to create a camera-profile. So if your equipment contains a target and you can create good images from them, you should be able to create a profile.

I already started to play with the images of the IT8 target from John, but the generated profiles did not show much difference to the color-matrix. The images are not optimal for profiling (skewed, overexposed) and they were never meant for that purpose. I think back in 2013 exposure compensation was also not available.

And keep in mind that the pi-camera is not a high end device, therefore this might be optimizing at the wrong end. And if color is your main target, you might even stick to jpg, since the jpg-engine does a very good job.

[tchiwam]

I'd be VERY happy to get the filters response (without the lens) !!!! 25nm resolution would be great even better if you can do both NoiR and normal filters. I could even buy and send you/give the missing unit if you don't have both.