waka
Posts: 3
Joined: Thu Apr 28, 2016 7:33 pm

Pi Camera V2 raw bayer data

Thu Apr 28, 2016 7:55 pm

Hi Guys,

Really appreciate the new camera module, already having a blast with it.

I have come across a problem with interpreting the raw bayer-data that is returned. Naturally, the format which is returned has changed from v1. Therefore, the instructions listed at http://picamera.readthedocs.io/en/relea ... bayer-data seem no longer valid.

My findings so far are that I can still find the header indicator ('BCRM') at byte 195539, but this leaves me with about 10237440 of 'raw' data after (which i hope is still) the 32768-byte long header. In v1 this data length was simply the number of pixels (rounded to 16 in x,y) of the sensor. However this no longer seems to be the case. Has the bit-depth changed e.g.? Anyone any ideas?

Regards,

WakA

User avatar
jbeale
Posts: 3439
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: Pi Camera V2 raw bayer data

Fri Apr 29, 2016 3:51 pm

Congratulations, you find yourself on the cutting edge of RPi camera development :-) AFAIK the V2 raw data format has not been documented yet. In v1 the information we have evolved through a slow process of reverse-engineering, because the official documentation was and still is under NDA.

If you would like to join the fun, you can make some guesses about the format, write some simple code, and see if the result forms an image which makes sense. That's what I did with the v1 camera anyway back in May 2013, see for example viewtopic.php?f=43&t=44918#p356571

To start with I'd note the IMX219 has (per http://www.sony.net/Products/SC-HP/IS/s ... 140910.pdf ) 3280 x 2464 (active pixels), 3296 x 2480 (effective pixels) and 3296 x 2512 (total pixels). If we assume the RAW mode reads out all pixels, active and masked that should be 8279552 pixels. If there are 10 bits per pixel in packed format, that occupies 10,349,440 bytes. If we look at only active pixels, at 10 bbp we get 10,102,400 pixels. Your number is in between those two values. The "Effective" pixel count of 3296 x 2480 => 10,217,600 bytes which is just 19,840 less than your value of 10,237,440. So if the RAW file has an extra 8 bytes of padding added to each of 2480 rows, that would give exactly the number you found.

I'm not sure that the IMX219 has a 10-bit ADC but I presume it does, if the image quality is at least as good as the v1 camera which was 10-bit.

User avatar
jbeale
Posts: 3439
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: Pi Camera V2 raw bayer data

Fri Apr 29, 2016 9:16 pm

If you want to share the joy, you could post the result of

Code: Select all

raspistill --raw -o raw.jpg
from the v2 camera somewhere for download, so others could have a look. It should be over 10 MB in size and it has to be the complete original file; most photo sharing sites will think you only care about the JPEG image, and downsize and resample the image and throw away the RAW data appended at the end.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6895
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Pi Camera V2 raw bayer data

Fri Apr 29, 2016 9:45 pm

Raw image size is 3280x2464 plus 2 pixels of padding on right and bottom, so 3282x2466.
Alignment on width is something like align up to 16, multiply by 10/8, align again to 16.
Alignment on height is probably to a multiple of 16

(edited after I went and read the source quickly)
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
jbeale
Posts: 3439
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: Pi Camera V2 raw bayer data

Fri Apr 29, 2016 11:18 pm

6by9 wrote:Raw image size is 3280x2464 plus 2 pixels of padding on right and bottom, so 3282x2466.
Alignment on width is something like align up to 16, multiply by 10/8, align again to 16.
Alignment on height is probably to a multiple of 16
That agrees with the block size reported:
Width: 3282 = 205.13 * 16, so 206 * 16 = 3296 and 10/8 * 3296 = 4120 which is 257.5 * 16, so 258 * 16 = 4128
Height: 2466 = 154.13 * 16, so 155 * 16 = 2480

4128 * 2480 = 10,237,440 bytes

by the way there is now an example v2 RAW file on my page at http://bealecorner.org/best/RPi/ (scroll to the bottom). I haven't had a chance to analyze it yet.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6895
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Pi Camera V2 raw bayer data

Sat Apr 30, 2016 6:22 am

Great - it was late so I didn't crunch the numbers last night. Have fun analysing the raws!
There's no change in the format of the 32kB header. The work that was done for determining Bayer order should all still apply.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

waka
Posts: 3
Joined: Thu Apr 28, 2016 7:33 pm

Re: Pi Camera V2 raw bayer data

Sat Apr 30, 2016 10:00 am

jbeale wrote:Congratulations, you find yourself on the cutting edge of RPi camera development :-)
Hah, yes, I had a sneaking suspicion. The thread you mentioned was somehow familiar to me :P Been using the info gleaned from that for a while now. Tnx!
jbeale wrote:If you want to share the joy, you could post the result of

Code: Select all

raspistill --raw -o raw.jpg
from the v2 camera somewhere for download, so others could have a look. It should be over 10 MB in size and it has to be the complete original file; most photo sharing sites will think you only care about the JPEG image, and downsize and resample the image and throw away the RAW data appended at the end.
I've made one available here. Captured using python with picamera.PiCamera.capture() and into a io.BytesIO() stream, then written to disk. Seems to be the same output as with raspistill.
6by9 wrote:Raw image size is 3280x2464 plus 2 pixels of padding on right and bottom, so 3282x2466.
Alignment on width is something like align up to 16, multiply by 10/8, align again to 16.
Alignment on height is probably to a multiple of 16

(edited after I went and read the source quickly)
Awesome. Thanks for the late-night info, I'll see if I can make it work. Meanwhile anyone else is more than welcome to the file above.

User avatar
Gavinmc42
Posts: 3172
Joined: Wed Aug 28, 2013 3:31 am

Re: Pi Camera V2 raw bayer data

Sat Apr 30, 2016 11:23 am

Interesting image properties for the 10.5MB file, 720x480, RP_OV5647.
Perhaps the picamera software needs updating for the V2:)

Bleeding edge, pack bandaids.

Would be interesting to try out the new QPU compiler running filters on the raw data.
Has anyone noticed that the Fathom Neural compute stick has 12 vector engines and so does the Pi GPU? http://www.movidius.com/
Tools run on ARM Linux, hmm, off track.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

waka
Posts: 3
Joined: Thu Apr 28, 2016 7:33 pm

Re: Pi Camera V2 raw bayer data

Sat Apr 30, 2016 12:27 pm

Okay, I figured it out it seems. The info given really helped.

Modifying the code from the docs gives us:

Code: Select all

	data = stream.getvalue()[-(10237440+32768):]
	assert data[:4] == 'BRCM'
	data = data[32768:]
	data = np.fromstring(data, dtype=np.uint8)
	full_sensor = False 
	if full_sensor:
		data = data.reshape((2480,4128))[:2466,:4125]
	else:
		data = data.reshape((2480,4128))[:2464,:4100]
the rest of the code can stay the same as far as I can tell.

There seems to be some interesting patterns on the side of the image when viewing the whole sensor. Not sure if they're random or if they have a purpose: AF/phase-contrast pixels, color calibration?
piv2_pixels.jpg
piv2_pixels.jpg (13.53 KiB) Viewed 13254 times

User avatar
Gavinmc42
Posts: 3172
Joined: Wed Aug 28, 2013 3:31 am

Re: Pi Camera V2 raw bayer data

Tue May 03, 2016 8:41 am

Have not had a look at the raw data yet.
I might need some fancy software.

Sony seems to indicate this is a RGBW chip, older parts say it.
The IMX219PQ has a lens shading correction function with four independent colors.
You may need to adjust the picamera code for this.
Not sure if the raw data from the V2 is RGBW or has been processed in the sensor to Bayer?

What does the 2x2 binning do to the raw data?
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

User avatar
jbeale
Posts: 3439
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: Pi Camera V2 raw bayer data

Tue May 03, 2016 2:13 pm

I do not believe this chip is RGBW. It is common for debayer algorithms to treat the two Green channels "RGGB" separately, and that is almost certainly what is meant by your quote.

For color images to work with 2x2 binning you have to bin like colors together, so you are not binning immediately adjacent pixels due to the bayer pattern. I don't know specifically the raw format used though; will probably have to be confirmed through trial & error- just takes some time.

User avatar
Gavinmc42
Posts: 3172
Joined: Wed Aug 28, 2013 3:31 am

Re: Pi Camera V2 raw bayer data

Wed May 04, 2016 4:03 am

Found some old Android source code
https://android.googlesource.com/kernel ... o/imx219.c

Done by Broadcom so they probably know how the chip works :oops:
Unless Sony has changed it since:)
The guy to ask = Guennadi Liakhovetski

Could be useful for raw data testing?
TEST_PATTERN_DISABLED,
https://gitgud.io/dankzegriefer/m8-GPE- ... r/imx219.c
https://github.com/ZenfoneArea/android_ ... i2c/imx219
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6895
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Pi Camera V2 raw bayer data

Wed May 04, 2016 7:03 am

Gavinmc42 wrote:Found some old Android source code
https://android.googlesource.com/kernel ... o/imx219.c

Done by Broadcom so they probably know how the chip works :oops:
Unless Sony has changed it since:)
The guy to ask = Guennadi Liakhovetski

Could be useful for raw data testing?
TEST_PATTERN_DISABLED,
https://gitgud.io/dankzegriefer/m8-GPE- ... r/imx219.c
https://github.com/ZenfoneArea/android_ ... i2c/imx219
The last two links may be useful for reference, but the qualcomm and Intel infrastructures are significantly different from the standard v4l2 one.

The Broadcom one (by Chethan Krishna and Alex Gladkov, both people I know from Broadcom days, but neither likely to still be there) has potential for fitting in with the raw capture stuff, although it uses the kernel SoC camera framework which is deprecated and dying. Conversion to a v4l2 subdevice isn't a huge job though. Doing something useful with all that data is another matter.

I really don't believe Sony is doing that much on chip processing, and the Broadcom isp doesn't support rgbw, so the sensor is going to be standard Bayer. Yes it is very common for the two greens to be treated independently, so sort of 4 channels.
Last edited by 6by9 on Tue Aug 02, 2016 9:55 am, edited 1 time in total.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
Gavinmc42
Posts: 3172
Joined: Wed Aug 28, 2013 3:31 am

Re: Pi Camera V2 raw bayer data

Wed May 04, 2016 8:15 am

Finding the references means the IMX219 has been used before.
Found at least 3 phones that use the IMX219.
With the Android reference, I think we can say the chip works ok.
And that it was/is a RGBG filter, standard bayer.

Grabbing the raws and running this over them would be another learning experience:)
http://rawtherapee.com/
Good to learn for budding photographers?
Rawtherapee running on 64bit OS RPi3?

Most image sensors are simple, costs money make complex ones.
Seen Sunny IMX219 modules for around $9, so sensor is probably $5?
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

User avatar
Gavinmc42
Posts: 3172
Joined: Wed Aug 28, 2013 3:31 am

Re: Pi Camera V2 raw bayer data

Thu May 05, 2016 4:14 am

A good explanation of what IR filters do and the extra colour correction needed.
Lens shading at the edges means less brightness as well.
Important when grabbing the RAW data.
Mostly putting this here because I did not know and someone might find it useful.

http://www.dxo.com/us/more-information- ... or-shading
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6895
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Pi Camera V2 raw bayer data

Mon May 09, 2016 5:06 pm

Happy Christmas - I've just hacked dcraw around to process the IMX219 raw images.
Source at https://github.com/6by9/RPiTest/blob/ma ... aw/dcraw.c. I have NOT rebuilt the binary on there as I was hacking on my x86 machine rather than the Pi.
There's no colour conversion matrix supplied, so the colours are well off at the moment. Feel free to take and modify, and once tidied up it's probably worth pushing back upstream to the main dcraw app.

Ideally I'd like to make it support searching for the @BRCM tag so that it can process raws from any of the Pi sensor modes, but one step at a time. (I wanted this to be able to analyse the lens issues without any ISP processing getting in the way).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
jbeale
Posts: 3439
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: Pi Camera V2 raw bayer data

Mon May 09, 2016 7:35 pm

6by9 wrote:I've just hacked dcraw around to process the IMX219 raw images. github.com/6by9/RPiTest/blob/master/dcraw/dcraw.c
Thank you! I can confirm that this compiles and works OK on a RPi. Color is as you say, a work in progress. Great stuff!

Code: Select all

wget https://raw.githubusercontent.com/6by9/RPiTest/master/dcraw/dcraw.c
gcc -o dcraw -O4 dcraw.c -lm -DNODEPS

wget http://bealecorner.org/best/RPi/PiCamV2-Raw-Chart2.jpg
./dcraw PiCamV2-Raw-Chart2.jpg
sudo apt-get install fbi
fbi PiCamV2-Raw-Chart2.ppm
Image

Koepi
Posts: 65
Joined: Fri Apr 29, 2016 7:18 am

Re: Pi Camera V2 raw bayer data

Sun May 15, 2016 8:09 am

Thank you for the "hacked" dcraw! :)

While I tried to fine tune the focus, I also took some raw images. Now that I snipped the raw part out, I could open the PPM with GIMP, save as PNG and use that in RawTherapee. I'm still getting used to that program, so bare with me. Here is the result of toying around with it - some vintage look, need to seriously improve my skills with that program:

Image

And here the JPG preview inside the original raw from raspistill (used jpg compression 95 to ensure no compression artefacts; file size is about the same as with the RawTherapee image):
Image


Direct comparison:
Image

A real good job in getting the colours right! But looking the RawTherapee picture, a few more details and a bit more sharpness should be possible to get. Looking at the lawn or the details of the fence mesh on the left.


In case you want to play around with it yourself, here is the complete RAW image (14MByte):
Image

User avatar
Gavinmc42
Posts: 3172
Joined: Wed Aug 28, 2013 3:31 am

Re: Pi Camera V2 raw bayer data

Sun May 15, 2016 8:24 am

Been using raw pics to check focus on the 8Mp V2 forum.
ImageMagick on a Linux box works with the --raw files fine.
RAW is better than jpg on the V2, FYI V1 raw is really good.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

Koepi
Posts: 65
Joined: Fri Apr 29, 2016 7:18 am

Re: Pi Camera V2 raw bayer data

Sun May 15, 2016 11:38 am

Gavin,
Did you strip the JPG "header" from your raspistill-raw picture? It has the whole JPG and just attaches the RAW data afterwards. Image software like GIMP or ImageMagick do some MIME magic and decode simply the JPG with which the file starts. You have to strip off the RAW data to use it! (Or need a program that knows about the "hidden" RAW data.)

User avatar
jbeale
Posts: 3439
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: Pi Camera V2 raw bayer data

Sun May 15, 2016 2:49 pm

Gavinmc42 wrote:Been using raw pics to check focus on the 8Mp V2 forum.
ImageMagick on a Linux box works with the --raw files fine.
To be more precise, ImageMagick works with the JPEG portion of the JPEG+RAW file that the raspistill -r option generates, and ignores the RAW part :-). Compile 6by9's version of dcraw to get a PPM file and compare, you will see what the noise reduction in the ISP is doing to get that JPEG image (the good, and the bad). The true raw image has a somewhat noisy texture when viewed at 1:1 pixels, even at the lowest ISO=50 setting. By the way be sure to use the dcraw -f option (Debayer: "Interpolate RGGB as four colors") otherwise you will get unpleasant pattern artifacts with this sensor.

naushir
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 10
Joined: Mon Apr 25, 2016 10:21 am

Re: Pi Camera V2 raw bayer data

Sun May 15, 2016 6:23 pm

Koepi wrote: Direct comparison:
Image

A real good job in getting the colours right! But looking the RawTherapee picture, a few more details and a bit more sharpness should be possible to get. Looking at the lawn or the details of the fence mesh on the left.
There may be some detail to be brought out, but not much. The RAW converted image in the comparison above shows lots of noise. Once you try to smooth that out, you will no doubt end up losing some of the finer detail visible in the grass and fence.

Koepi
Posts: 65
Joined: Fri Apr 29, 2016 7:18 am

Re: Pi Camera V2 raw bayer data

Sun May 15, 2016 6:52 pm

Of course! That's why I wrote "a few" and "a bit". It's not much, this is clear.
Maybe raspistill could get an option for setting denoising level low/normal or something along those lines. Or is --sharpness already doing this?

User avatar
puppy101puppy
Posts: 16
Joined: Mon Feb 02, 2015 8:19 am

Re: Pi Camera V2 raw bayer data

Tue Jun 28, 2016 4:08 am

jbeale wrote:
6by9 wrote:I've just hacked dcraw around to process the IMX219 raw images. github.com/6by9/RPiTest/blob/master/dcraw/dcraw.c
Thank you! I can confirm that this compiles and works OK on a RPi. Color is as you say, a work in progress. Great stuff!

Code: Select all

wget https://raw.githubusercontent.com/6by9/RPiTest/master/dcraw/dcraw.c
gcc -o dcraw -O4 dcraw.c -lm -DNODEPS

wget http://bealecorner.org/best/RPi/PiCamV2-Raw-Chart2.jpg
./dcraw PiCamV2-Raw-Chart2.jpg
sudo apt-get install fbi
fbi PiCamV2-Raw-Chart2.ppm
Image
Hi Jbeale,

Thanks for sharing -- indeed it compiles and works for your image.
However, I tried with my raw images captured with raspberry pi camera v2 and it generates a random noise raw image (so there's something wrong with encoding). I was using the command below to capture raw images:

Code: Select all

 raspistill -n -r -o filename.jpg 
Are you also using camera v2? What is your command line to capture your raw image, PiCamV2-Raw-Chart2.jpg?

Also, "dcraw" function it generates a warning, although it creates the raw image fine for your PiCamV2-Raw-Chart2.jpg (not the raw image captured with my above command):

Code: Select all

 ./dcraw filename.jpg filename.ppm 
cannot decode file filename.ppm 
Help appreciated.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6895
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Pi Camera V2 raw bayer data

Tue Jun 28, 2016 7:26 am

puppy101puppy wrote:Thanks for sharing -- indeed it compiles and works for your image.
However, I tried with my raw images captured with raspberry pi camera v2 and it generates a random noise raw image (so there's something wrong with encoding). I was using the command below to capture raw images:

Code: Select all

 raspistill -n -r -o filename.jpg 
Are you also using camera v2? What is your command line to capture your raw image, PiCamV2-Raw-Chart2.jpg?
Can you post one of your files somewhere for analysis? dcraw was working fine when I made the mods.
puppy101puppy wrote:Also, "dcraw" function it generates a warning, although it creates the raw image fine for your PiCamV2-Raw-Chart2.jpg (not the raw image captured with my above command):

Code: Select all

 ./dcraw filename.jpg filename.ppm 
cannot decode file filename.ppm 
dcraw automatically creates the output filename from the source one, so your command is trying to process the two files filename.jpg and filename.ppm as two raw files.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Return to “Camera board”