RAW output information

[Schwabenchris]

I'd also say avoid the really long exposure time mode

Need the long exposure time to measure it's dark noise correctly.
I'm doing this for my bachelor thesis.

You can see the amount of dark noise is getting significantly higher with longer exposure times, if you compare the histogram of my shot with the histogram of jbeale's shot.

blindbloom:
Is your statement for all raw images or just for the raw-data from the raspberry pi? Wouldn't make sense to change it in Matlab/Mathematica if it's a raspi specific thing.

Btw: that's the histogram I would expect, poison-distributed or "Gaussian"-alike! My dark noise simulation in Matlab looks very similar.

[6by9]

Ok. I figured this all out. The (mostly undocumented) raw data is appended to the end of the JPEG file, as described elsewhere. The pixels are 10-bits each and four sequential pixels are encoded in five sequential bytes. This is the correct format:
<snip>

That information had been provided ages ago. viewtopic.php?f=43&t=44918&start=75#p575853 for one gave you a link to the CCP2 spec, though the quoted link seems to be a dead now. It's also been stated multiple times in threads that you get 4 bytes with the MSBits, and then a fifth byte that is the LSBits of those 4 pixels. Ho hum.

The Mathematica and Matlab decoders should be changed accordingly. Maybe Schwabenchris can contact them with this info?

raspi_dng appears to correctly convert from 10bit raw to 16bit - https://github.com/illes/raspiraw/blob/ ... dng.c#L166
The original post by Schwabenchris said they were using raspi_dng, so I'm confused why you're wanting Mathematica or Matlab to be doing anything other than handle 16bit data. I must confess to not being convinced by the settings for the bits/pixel in the TIFF headers that raspi_dng is writing out.

Anyway, appears to be PEBKAC, or misuse of tools, so I'm not worried about it.

[blindbloom]

I guess the bottom line is that the originally posted problem has now been solved (despite the involvement of imperfect individuals like myself). I'm glad for that.

[Schwabenchris]

Thanks for the input!

But where is the point, the wrong decoding happens? rpi_dng?

[blindbloom]

Schwabenchris provided two DNG files:

• darkshot.dng (raspiraw, Software tag = "raspi_dng")

• darkshot+raw.dng (rpi2dng, Software tag = "raspi2dng")

I plotted codes [0, 31] from the entire [0, 65535] range for G1:

DNG_Comparison.png (12.13 KiB) Viewed 11730 times

Observations:

• The pixel content of the two is similar, but different.

• The values have been converted to 16-bit, since that largest value found is ~49,000.

• The conversion to 16-bit appears to be non-linear, since the codes for the dark end of the scale are compressed, while the light end is expanded.

• The histograms do not show the "stepped" shape, although this effect may be masked due to the nature of the conversion. I cannot explain how the histograms of these files show stepping when imported by Mathematica or Matlab.

Yes, there is the possibility that I've made errors in this analysis. I suggest the analysis be repeated by someone else who is more competent than me. I am only trying to help, so please don't PEBKAC me.

My suggestion is that if noise metrics are to be measured, data closest to the sensor should be used. IOW, I suggest that the JPEG file containing the raw appendage contains more trustworthy raw data than DNG files that have had unknown data processing applied to the raw sensor data.

[sharix]

I wrote a function to import raw images in Mathematica with proper 10-bit decoding. It's pretty fast, creating the bayer pixel array in less than 0.4 seconds. Converting the array to image takes 0.01 seconds, and demosaicing takes additional 5 seconds.

Code: Select all

c = Compile[{{cbl, _Integer, 1}, {coffset, _Integer}},
  Module[{cpixel = Table[0, {1944}, {2592}], row, j, split, col, 
    buffer},
   For[row = 0, row < 1944,
    buffer = cbl[[coffset + row*3264 ;; coffset + (row + 1)*3264 - 1]];
    j = 0;
    row++;
    For[col = 0, col < 2592,
     cpixel[[row, col + 1]] = buffer[[++j]]*2^8;
     cpixel[[row, col + 2]] = buffer[[++j]]*2^8;
     cpixel[[row, col + 3]] = buffer[[++j]]*2^8;
     cpixel[[row, col + 4]] = buffer[[++j]]*2^8;
     split = buffer[[++j]];
     cpixel[[row, col + 1]] += BitAnd[split, 192];
     cpixel[[row, col + 2]] += BitAnd[split, 48]*2^2;
     cpixel[[row, col + 3]] += BitAnd[split, 12]*2^4;
     cpixel[[row, col + 4]] += BitAnd[split, 3]*2^6;
     col += 4;
     ]
    ];
   cpixel
   ]
  , CompilationOptions -> {"InlineCompiledFunctions" -> True, 
    "InlineExternalDefinitions" -> True}, CompilationTarget -> "C", 
  "RuntimeOptions" -> "Speed"]

(* Bayer pixel array *)
bl = BinaryReadList["darkshot+raw.jpg", "Byte"];
offset = FileByteCount["darkshot+raw.jpg"] - 6404096 + 32768 + 1;
pixels = c[bl, offset];

(* Bayer array image *)
imagec = Image[pixels/2.^16];

(* debayered image *)
imaged = ImageDemosaic[imagec, {"RGGB", {1, 0}}];

The debayered image histogram looks like this:

[Schwabenchris]

Matlab code:

Code: Select all

filename='filename.jpg'
sizeRAW = 6404096;
sizeHeader =32768;
imagewidth=2592;
imageheight=1944;
stepsize=4;
rowsize =3264;                                                          %number of bytes per row of pixels, including 24 'other' bytes at end

fin=fopen(filename,'r');                                                
off1 = dir(filename);                                                   %get fileinfos
offset = off1.bytes - sizeRAW + sizeHeader;                             %calc the offset for the reading process
fseek(fin, offset,'bof');                                               %
pixel = uint16(ones(imagewidth,imageheight));
	for row=1:imageheight 
		I=fread(fin,rowsize,'ubit8','ieee-le');
		x = 0;
		for col=1:stepsize:imagewidth
		    x = x+1;  
		    pixel(col+0,row) = I(x)*2^8;
		     x = x+1;
		      pixel(col+1,row) = I(x)*2^8;
		       x = x+1;
		      pixel(col+2,row) = I(x)*2^8;
		       x = x+1;
		      pixel(col+3,row) = I(x)*2^8;
		       x = x+1;
		      split = I(x);                                                 %grab the LSB byte
		      pixel(col+0,row) = pixel(col+0,row) + bitand(split,192);      %split up the LSB byte with 0b11000000 aka 192 and add to pixel 1
		      pixel(col+1,row) = pixel(col+1,row) + bitand(split,48)*2^2;   %split up the LSB byte with 0b00110000 and add to pixel 2
		      pixel(col+2,row) = pixel(col+2,row) + bitand(split,12)*2^4;   %split up the LSB byte with 0b00001100 and add to pixel 3
		      pixel(col+3,row) = pixel(col+3,row) + bitand(split,3)*2^6;    %split up the LSB byte with 0b00000011 and add to pixel 4
		end
	end

[pfile]

regarding the bayer matrix - i can see that the DNG files written out by raspi_dng have "Filter Pattern: BG/GR" as announced by dcraw.

however, on my omnivision NoIR camera module, as far as i can tell the filter pattern needs to be GB/RG (established by taking a daylight photo thru an IR cut filter). otherwise, dcraw decodes the file with crazy colors.

is this a difference between the NoIR camera and the standard camera, or do i have some kind of endianness problem or...? this code has been around a while so i feel like the problem must be on my end.

thanks

[6by9]

regarding the bayer matrix - i can see that the DNG files written out by raspi_dng have "Filter Pattern: BG/GR" as announced by dcraw.

however, on my omnivision NoIR camera module, as far as i can tell the filter pattern needs to be GB/RG (established by taking a daylight photo thru an IR cut filter). otherwise, dcraw decodes the file with crazy colors.

is this a difference between the NoIR camera and the standard camera, or do i have some kind of endianness problem or...? this code has been around a while so i feel like the problem must be on my end

Have you used a horizontal or vertical flip? That alters bayer order as it just reverses the direction the sensor arrays read out.
I can't remember what the default order is but it did change back in about 2013 due to an error in the config.

[pfile]

i haven't specified a flip - the image orientation looks OK as is - the CFA file has same orientation as the jpeg that's produced and seems to be "right side up". at any rate hacking raspi_dng to change the filter orientation tag in the dng header seems to work; dcraw debayers the image with proper colors after the hack.

unrelated question - is 5.9s a hardware limitation for the max exposure? now that i am able to calibrate the CFA images with darks the thermal signal is not that much of a problem. longer exposures would be nice for my silly astrophotography projects.

any idea if the sony-based camera module can expose longer?

thanks

[6by9]

i haven't specified a flip - the image orientation looks OK as is - the CFA file has same orientation as the jpeg that's produced and seems to be "right side up". at any rate hacking raspi_dng to change the filter orientation tag in the dng header seems to work; dcraw debayers the image with proper colors after the hack.

Looking at https://github.com/illes/raspiraw, raspi_dng hasn't been updated since 2 June 2013. Checking the firmware history, the Bayer order was fixed on 22 May 2013, so I doubt raspi_dng takes that into account.
Why are you worrying about raspi_dng anyway? dcraw has been able to natively process the JPEG+raw files produced by Pi since at least 1.470, which appears to be 13 Feb 2015.
If you grab my hacked version of dcraw from https://github.com/6by9/RPiTest/tree/master/dcraw, then that version has been updated for the V2.1 camera and processing the raw header to get the Bayer order correct with any flip combination. (Ignore the binary in that directory - it didn't get updated, but jbeale rebuilt the image as noted on viewtopic.php?f=43&t=146310#p970933)

unrelated question - is 5.9s a hardware limitation for the max exposure? now that i am able to calibrate the CFA images with darks the thermal signal is not that much of a problem. longer exposures would be nice for my silly astrophotography projects.

any idea if the sony-based camera module can expose longer?

I'm fairly certain it has been stated quite a few times that the Sony sensor will currently go up to 10 second exposures.
It's a semi-hardware limitation in that that is the maximum exposure time we managed to achieve on either sensor. Trying to tweak it further resulted in corrupt frames being received. OV claimed they had 20 second exposures working, but I never managed to lock on to them.

[pfile]

Looking at https://github.com/illes/raspiraw, raspi_dng hasn't been updated since 2 June 2013. Checking the firmware history, the Bayer order was fixed on 22 May 2013, so I doubt raspi_dng takes that into account.

ah - that's the issue.

Why are you worrying about raspi_dng anyway? dcraw has been able to natively process the JPEG+raw files produced by Pi since at least 1.470, which appears to be 13 Feb 2015.

d'oh!! i think i'm suffering from backwater system syndrome - it would appear that on both my systems (the rpi and osx (macports) the dcraw version is severely downrev. i had no idea that dcraw knew how to parse the jpeg+raw!

If you grab my hacked version of dcraw from https://github.com/6by9/RPiTest/tree/master/dcraw, then that version has been updated for the V2.1 camera and processing the raw header to get the Bayer order correct with any flip combination. (Ignore the binary in that directory - it didn't get updated, but jbeale rebuilt the image as noted on viewtopic.php?f=43&t=146310#p970933)

i will grab this, thanks!

I'm fairly certain it has been stated quite a few times that the Sony sensor will currently go up to 10 second exposures.
It's a semi-hardware limitation in that that is the maximum exposure time we managed to achieve on either sensor. Trying to tweak it further resulted in corrupt frames being received. OV claimed they had 20 second exposures working, but I never managed to lock on to them.

ok thanks for that, the thread is really long and i missed it. i think i will start playing around with one of the sony modules and see how things look. some of their larger sensors have extremely low dark current; would be interesting to see if the cellphone sensors are similar.

[blindbloom]

I am also interested in long exposures and have heard that this leads to heat problems. Is that true? If so, does that lead to more noise? Does it lead to permanent H/W damage?

[6by9]

I am also interested in long exposures and have heard that this leads to heat problems. Is that true? If so, does that lead to more noise? Does it lead to permanent H/W damage?

citation and new thread needed. It's not related to raw output.

[blindbloom]

I have a couple questions about RAW output behavior, and since this thread is titled "Raw Output Information" it seems like a great place for me to learn from others who are willing to share their knowledge and experience.

I am wondering about heat-related issues when taking RAW images with long exposure times. Specifically, has anyone attempted to correlate the noise characteristics in the RAW output with the temperature of the sensor? One application example that comes to mind is maybe someone working on silly astrophotography projects . A quick look at Wikipedia under astrophotography shows that heat buildup can be a real problem with time exposures and sequential time exposures. Has anyone experienced component damage due to this heat buildup? Say, if you took time exposures all night long.

What is the relevance to RAW output specifically? Everything! The only way that true noise metrics of the sensor may be obtained is through the RAW appendage to the JPEG image. Noise analysis using a JPEG image is wrong for many reasons. JPEG images (from cameras in general, including RPi cameras) have typically been processed in ways that change the noise characteristics (demosaicing, matrix, tone curves, noise-reduction, sharpening). JPEG format itself is lossy due to the type of compression algorithm used. Further, since JPEG stores image color in Y Cb Cr format (not RGB) as three bytes, there is a bit-depth reduction on top of the fact that the Y Cb Cr gamut volume is only 23.628% the size of the corresponding RGB gamut volume.

By the time a RAW image is placed into JPEG format, the noise characteristics of the raw sensor are seriously obfuscated.

I would greatly appreciate insights from any kind folks having a desire to help others. Thanks in advance.

[jbeale]

The sensor temperature is related to ambient temperature, and the total electrical power used which generates heating above ambient. As far as I know, running the RPi camera sensor continually in long-exposure mode does not use more total power than running the sensor continually at a normal frame rate like 25 or 30 fps. In fact I would expect it would take less power, as there is less high-frequency switching from the readout going on.

I have several v1 and v2 cameras running in video mode continually, 24/7 and I have not observed any noise problem or reliability problem with this use. I am not using RAW output, but with the RPi camera that is on the ISP side, the sensor is always generating raw output regardless of mode setting so it's the same thing from the sensor point of view.

My expectation is that sensor noise will go up with temperature, and that's why astronomy sensors are generally cooled, but AFAIK professional astronomy is always done with CCD sensors, not CMOS as both v1 and v2 RPi cameras use. Here is a PhD thesis about noise in CMOS sensors: http://www-isl.stanford.edu/~abbas/grou ... thesis.pdf

[blindbloom]

Very helpful. Thank you so much!

[6by9]

I have a couple questions about RAW output behavior, and since this thread is titled "Raw Output Information" it seems like a great place for me to learn from others who are willing to share their knowledge and experience.

The original request in this thread was "I'm curious in the raw output option from the raspistill utlilty." and this thread is almost exclusively about how to process that data. Your request is specific to how the sensor itself works.

The forum as a whole is a great place to learn, but if you go and take threads significantly off-topic then it makes it almost impossible to find information at a later date - you're on page 9 of this thread. You may also find that people would NOT read this thread if they aren't worried about raw output, but they may have knowledge of how the sensor works - thread title is important.
My recommendation (OK maybe it was a little terse) was that you start a new thread to discuss the subject, but I guess this will just get lost in the noise instead. I'm happy that you have your answer, but I doubt others will be able to benefit in future due to being buried. Ho hum.

[blindbloom]

Yes, in principle I agree with you, but forums are, by nature, notorious for digressions and bifurcations. Nevertheless, I will be more mindful of your guidelines in the future. Thank you for pointing this out.

[pfile]

Why are you worrying about raspi_dng anyway? dcraw has been able to natively process the JPEG+raw files produced by Pi since at least 1.470, which appears to be 13 Feb 2015.
If you grab my hacked version of dcraw from https://github.com/6by9/RPiTest/tree/master/dcraw, then that version has been updated for the V2.1 camera and processing the raw header to get the Bayer order correct with any flip combination. (Ignore the binary in that directory - it didn't get updated, but jbeale rebuilt the image as noted on viewtopic.php?f=43&t=146310#p970933)

i now have a sony sensor and using your hacked dcraw i can create tiffs from the sony raw+jpg. however, when i run this version on an omnivision raw+jpeg, the output is corrupt.

i compiled stock dcraw 1.477 (which looks like its from 10 May 2016), and it apparently can not parse the omnivision jpg+raw. stepping thru the code, i can see that in identify(), dcraw is looking for the string "BRCMn" at the start of the raw data, but apparently my files have "BRCMo" in the raw data. changing that string makes it work, and it produces valid tiff files. the camera is the noir v1.3 module. not sure if that has anything to do with this.

[6by9]

i now have a sony sensor and using your hacked dcraw i can create tiffs from the sony raw+jpg. however, when i run this version on an omnivision raw+jpeg, the output is corrupt.

i compiled stock dcraw 1.477 (which looks like its from 10 May 2016), and it apparently can not parse the omnivision jpg+raw. stepping thru the code, i can see that in identify(), dcraw is looking for the string "BRCMn" at the start of the raw data, but apparently my files have "BRCMo" in the raw data. changing that string makes it work, and it produces valid tiff files. the camera is the noir v1.3 module. not sure if that has anything to do with this.

I had noticed that but wasn't worrying about OV5647 as no one had complained, and support of 3rd party apps is a never ending saga.
The 'n' or 'o' is the header version field so shouldn't really be used in identify in the string search at all. It got bumped in June 2015 when there was a fix for writing raw headers on stereoscopic images.

[6by9]

i now have a sony sensor and using your hacked dcraw i can create tiffs from the sony raw+jpg. however, when i run this version on an omnivision raw+jpeg, the output is corrupt.

i compiled stock dcraw 1.477 (which looks like its from 10 May 2016), and it apparently can not parse the omnivision jpg+raw. stepping thru the code, i can see that in identify(), dcraw is looking for the string "BRCMn" at the start of the raw data, but apparently my files have "BRCMo" in the raw data. changing that string makes it work, and it produces valid tiff files. the camera is the noir v1.3 module. not sure if that has anything to do with this.

I had noticed that but wasn't worrying about OV5647 as no one had complained, and support of 3rd party apps is a never ending saga.
The 'n' or 'o' is the header version field so shouldn't really be used in identify in the string search at all. It got bumped in June 2015 when there was a fix for writing raw headers on stereoscopic images.

Looking at the code again, I see I did the right thing for IMX219. I've amended OV5647 to do the same and only look for the 4 char string - pushed to the same github repo.

[pfile]

Looking at the code again, I see I did the right thing for IMX219. I've amended OV5647 to do the same and only look for the 4 char string - pushed to the same github repo.

ok i picked it up, thanks, now it can decode my omnivision and sony files to tiff properly. goodbye to raspi_dng!

[Hodapp87]

The format from the OV5647-based Arducam boards seems to have changed again, as I just picked up a new one from Amazon in the past month, and I had to patch current dcraw to handle images from it.

I just posted at viewtopic.php?p=1043088#p1043088 about it.

[Bryan See]

Came by to see this thread, I'll comment around a bit. I tried the first example Python script of the Bayer capture, but I still cannot open the image file generated at the end of it, as the file is supposed to be readable in most packages (like GIMP and Photoshop). The question is: Could that be in DNG format? Or something else?

BTW, I've extracted the bayer data from the Jpeg stream or file, whether it's V1 or V2.