bablokb
Posts: 25
Joined: Fri Nov 07, 2014 7:45 am

Re: RAW output information

Fri Nov 28, 2014 3:12 pm

6by9 wrote:So those are the CCMs and AWB gains that the internal algorithms have chosen. Half the point of capturing raws is that you can tinker with them, as otherwise you are just going to recreate the same image as the GPU (although you could tweak values in other parts of the processing chain).
The other half of the point is that you have a good starting value for optimizing colors (unless the camera is completely off). Every decent raw converter offers the choice to set AWB values to the values the camera has choosen - if they are available.

Independently of this argument, you just need a color-matrix. Raspiraw just uses an identity-matrix which is way off, while dcraw uses a hard-coded default. Not too bad for average light situations, but not every situation has average light.

User avatar
jbeale
Posts: 3500
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: RAW output information

Fri Nov 28, 2014 5:24 pm

I have updated http://bealecorner.org/best/RPi/ to include a RAW sample from raspistill using November 2014-era firmware.

Supposedly this color matrix is reasonable: http://www.raspberrypi.org/forums/viewt ... 50#p511476

tchiwam
Posts: 43
Joined: Mon Nov 24, 2014 4:01 pm

Re: RAW output information

Sat Nov 29, 2014 1:26 am

This seems to be the most neutral and fastest back to back raw I can have

raspistill -v -set -n -bm --raw -th none -awb off -awbg 1.0,1.0 -drc off -tl 0 --quality 100 -t 43200000 -ss 6000000 -ISO 800 -l latest.jpg -o `date +%Y%m%d-%H%M%S`-12H-ISO800-6000ms-%05d.jpg

2014-11-29 01:13:51.133117356 +0000 20141129-003315-12H-ISO800-6000ms-00193.jpg
2014-11-29 01:14:03.732839072 +0000 20141129-003315-12H-ISO800-6000ms-00194.jpg
2014-11-29 01:14:16.282561936 +0000 20141129-003315-12H-ISO800-6000ms-00195.jpg
2014-11-29 01:14:28.782285936 +0000 20141129-003315-12H-ISO800-6000ms-00196.jpg
2014-11-29 01:14:41.322009108 +0000 20141129-003315-12H-ISO800-6000ms-00197.jpg
2014-11-29 01:14:53.811733416 +0000 20141129-003315-12H-ISO800-6000ms-00198.jpg
2014-11-29 01:15:06.391455789 +0000 20141129-003315-12H-ISO800-6000ms-00199.jpg

Then I'll use exiftool to get the creation time, would be nice to add the exif timestamp of t0 and scan clock ;)

bablokb
Posts: 25
Joined: Fri Nov 07, 2014 7:45 am

Re: RAW output information

Sat Nov 29, 2014 10:38 am

jbeale wrote:I have updated http://bealecorner.org/best/RPi/ to include a RAW sample from raspistill using November 2014-era firmware.

Supposedly this color matrix is reasonable: http://www.raspberrypi.org/forums/viewt ... 50#p511476
Thanks a lot, I will check the new image this afternoon. The color matrix from the quote is actually the hard-coded default from dcraw and it is indeed reasonable (at least for outdoor shots).

User avatar
jbeale
Posts: 3500
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: RAW output information

Sat Nov 29, 2014 8:00 pm

I got dcraw.c to work on new & old RAW images without needing to use exiftool. See also: http://www.raspberrypi.org/forums/viewt ... 59#p647559

bablokb
Posts: 25
Joined: Fri Nov 07, 2014 7:45 am

Re: RAW output information

Sun Nov 30, 2014 2:11 pm

jbeale wrote:I got dcraw.c to work on new & old RAW images without needing to use exiftool. See also: http://www.raspberrypi.org/forums/viewt ... 59#p647559
I have updated the raspi_dng code (see https://github.com/bablokb/raspiraw). This version also accepts a color-matrix as an optional third parameter. The new wrapper script raspi_dng.sh extracts the matrix from the EXIF-data and then calls raspi_dng for the actual conversion using this matrix.

I still have to gain some insights into this matrix though. Scaling seems technically not necessary, but scaling the values down to around 1 gives better results (at least from first experiments and using rawtherapee to process the dng-file).

tchiwam
Posts: 43
Joined: Mon Nov 24, 2014 4:01 pm

Re: RAW output information

Sun Nov 30, 2014 3:23 pm

Are the raw values going to be changed or is this matrix is stored so that the display program knows what to do with the data ?

User avatar
jbeale
Posts: 3500
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: RAW output information

Sun Nov 30, 2014 5:05 pm

bablokb wrote:I have updated the raspi_dng code (see https://github.com/bablokb/raspiraw). This version also accepts a color-matrix as an optional third parameter. The new wrapper script raspi_dng.sh extracts the matrix from the EXIF-data and then calls raspi_dng for the actual conversion using this matrix.

I still have to gain some insights into this matrix though. Scaling seems technically not necessary, but scaling the values down to around 1 gives better results (at least from first experiments and using rawtherapee to process the dng-file).
Good work. I'm not sure I understand color matrix operations, but one difference with scaling is if the values are large, color values might get clipped during intermediate computations or at output. If they are too small, might loose precision due to roundoff. I don't know if those are done in floating-point and then rescaled, or (faster but more dangerous) in fixed-point format.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23946
Joined: Sat Jul 30, 2011 7:41 pm

Re: RAW output information

Sun Nov 30, 2014 5:07 pm

Internally, IIRC, the ISP does calculations in 16p16 fixed point.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7456
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: RAW output information

Sun Nov 30, 2014 7:05 pm

jamesh wrote:Internally, IIRC, the ISP does calculations in 16p16 fixed point.
My recollection is the same. Although they are generally saturating arithmetic operations so you don't get odd wrap arounds in values. Floating point is just too expensive on power, silicon area, and generally time to be considered viable (the ISP in BCM2835 is processing a pixel per clock cycle. Later versions could do up to 4 pixels/clock).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

bablokb
Posts: 25
Joined: Fri Nov 07, 2014 7:45 am

Re: RAW output information

Mon Dec 01, 2014 6:30 pm

tchiwam wrote:Are the raw values going to be changed or is this matrix is stored so that the display program knows what to do with the data ?
Raspiraw just stores the matrix in a TIFF-header field. The converter/display-program has to take care of the data.

tchiwam
Posts: 43
Joined: Mon Nov 24, 2014 4:01 pm

Re: RAW output information

Tue Dec 02, 2014 11:46 am

-awbg 1.0,1.0

If this analog gain on the R and B channel or digital multiplier ? And does it in anyway affect the raw data ?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7456
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: RAW output information

Tue Dec 02, 2014 12:29 pm

tchiwam wrote:-awbg 1.0,1.0

If this analog gain on the R and B channel or digital multiplier ?
Digital multiplier. I haven't come across any sensor where you can set different gains for the different channels.
tchiwam wrote:And does it in anyway affect the raw data ?
Nope. Those gains are applied in the BCM2835 ISP.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

ethanol100
Posts: 587
Joined: Wed Oct 02, 2013 12:28 pm

Re: RAW output information

Tue Dec 02, 2014 12:47 pm

Just a small remark: If you set values which do not make much sense(i.e., 0.001,0.001). The auto exposure algorithm can than change the analog gain, which would change the raw data.

aphextwin
Posts: 10
Joined: Tue Dec 02, 2014 5:46 pm

Re: RAW output information

Tue Dec 02, 2014 5:51 pm

hi there,

I have problems with the raspi_dng:
I'm getting compilation-errors when trying to compile on rip and on a ubuntu 64 bit system.
Can somebody please upload a compiled binary of the raspi_dng ??


thanks a lot!

User avatar
jbeale
Posts: 3500
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: RAW output information

Tue Dec 02, 2014 9:04 pm

aphextwin wrote:I have problems with the raspi_dng:
I'm getting compilation-errors when trying to compile on rip and on a ubuntu 64 bit system.
If you specify what specific errors you see, and which source you are using, eg. is it the newer
https://github.com/bablokb/raspiraw or the code from last year https://github.com/illes/raspiraw
...then there is a better chance of solving the issue.

aphextwin
Posts: 10
Joined: Tue Dec 02, 2014 5:46 pm

Re: RAW output information

Wed Dec 03, 2014 8:16 am

sorry, you're absolutely right.

I tried to compile the newer version on my rpi b+.
The Illes-version compiled like a charm but the newer did not work at the very end. It seemed a tiff-library was missing.
I'll get the compile-log later, I don't have it right now.

What I try to do is using Imatest, of which I own a license, to measure MTF of different lenses.

I kind of solved my issue right now by writing the new rgb-values into the old Illes-code, which works fine by now.

Anyways I wanted to thank you all for your posts in this thread which are a great contribution!!

bablokb
Posts: 25
Joined: Fri Nov 07, 2014 7:45 am

Re: RAW output information

Thu Dec 04, 2014 7:49 pm

aphextwin wrote: I have problems with the raspi_dng:
I'm getting compilation-errors when trying to compile on rip and on a ubuntu 64 bit system.
As stated in the readme, compiliation on rip is currently broken. I didn't have access to my pi for a couple of days, I will fix that soon. Setting LIBSUFFIX to empty probably should do the trick. This is just a hack, I have to figure out something better (automake or cmake just seems overkill for such a small program).
aphextwin wrote: Can somebody please upload a compiled binary of the raspi_dng ??
I added a subdirectory 'bin' to my repository https://github.com/bablokb/raspiraw with a x64-binary of raspi_dng. This might or might not run on your system.

aphextwin
Posts: 10
Joined: Tue Dec 02, 2014 5:46 pm

Re: RAW output information

Fri Dec 05, 2014 9:51 am

Thanks, I just the old one in which I altered the rgb-values. Take your time.

bablokb
Posts: 25
Joined: Fri Nov 07, 2014 7:45 am

Re: RAW output information

Fri Dec 05, 2014 12:45 pm

aphextwin wrote:Thanks, I just the old one in which I altered the rgb-values. Take your time.
I fixed the makefile, it works again on the RPI. I also added the binary I just compiled to the bin directory of https://github.com/bablokb/raspiraw.

aphextwin
Posts: 10
Joined: Tue Dec 02, 2014 5:46 pm

Re: RAW output information

Mon Dec 08, 2014 7:41 am

Thanks, I'll try to compile it when I'm at the office and give a feedback.

tchiwam
Posts: 43
Joined: Mon Nov 24, 2014 4:01 pm

Re: RAW output information

Tue Dec 09, 2014 3:40 am

Just in local do this:
$ cd local
$ ln -s lib lib64

gcc: error: local/lib64/libtiff.a: No such file or directory
Makefile:32: recipe for target 'raspi_dng' failed
make: *** [raspi_dng] Error 1

aphextwin
Posts: 10
Joined: Tue Dec 02, 2014 5:46 pm

Re: RAW output information

Tue Dec 09, 2014 4:45 pm

Hi,

the new raspi_dng-binary works on my rpi, thanks for that.

As a student I was able to get my hands on a Imatest-Master-License and at my lab we've also got a spectrophotometer with which I can measure Lab and XYZ-values.
What I'd like to know is how can I refine the rgb-matrix with Imatest as I am new to this program? Of course I will share my results so we all get better images.

bablokb
Posts: 25
Joined: Fri Nov 07, 2014 7:45 am

Re: RAW output information

Tue Dec 09, 2014 7:23 pm

[quote="aphextwin"
What I'd like to know is how can I refine the rgb-matrix with Imatest as I am new to this program? Of course I will share my results so we all get better images.[/quote]

No idea about imatest, but in http://ninedegreesbelow.com/photography ... ofile.html you will find a discussion on how to create a camera-profile. So if your equipment contains a target and you can create good images from them, you should be able to create a profile.

I already started to play with the images of the IT8 target from John, but the generated profiles did not show much difference to the color-matrix. The images are not optimal for profiling (skewed, overexposed) and they were never meant for that purpose. I think back in 2013 exposure compensation was also not available.

And keep in mind that the pi-camera is not a high end device, therefore this might be optimizing at the wrong end. And if color is your main target, you might even stick to jpg, since the jpg-engine does a very good job.

tchiwam
Posts: 43
Joined: Mon Nov 24, 2014 4:01 pm

Re: RAW output information

Tue Dec 09, 2014 9:49 pm

I'd be VERY happy to get the filters response (without the lens) !!!! 25nm resolution would be great even better if you can do both NoiR and normal filters. I could even buy and send you/give the missing unit if you don't have both.

Return to “Camera board”