GauVeldt
Posts: 24
Joined: Thu Mar 14, 2013 4:46 pm
Contact: Website

Re: Camera Interface Specs

Sat Mar 16, 2013 1:48 am

jamesh wrote:Image System Pipeline is one definition (there are others). It's the path the image takes from being captures by the sensor to being fully processed and decent. On the GPU there are >20 stages to the ISP. The CSI-2 port where the camera attaches is just the first stage. That produces a raw bayer image, which is bebayered, black level corrected, lens shading corrected, denoised, sharpened, colour balanced, runs through the gain control, the white balance control. And that's just a subset of the processing...all done on the GPU at 30fps for 1080p.

Note that some camera sensors have all the processing built in - in fact the one being used does, but we don't actually use the internal ISP in this case, we use the one on the GPU, and just take bayer data from the camera.
It still sounds like many of the processes mentioned above could be done by cleverly written fragment shaders (render-to-texture) if public interfaces to use GLSL or Cg exist.
jamesh wrote:ref: You may be right should you 'you are right'
<<<Attack packet dropped by firewall>>>

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 25393
Joined: Sat Jul 30, 2011 7:41 pm

Re: Camera Interface Specs

Sat Mar 16, 2013 8:38 am

GauVeldt wrote:
jamesh wrote:Image System Pipeline is one definition (there are others). It's the path the image takes from being captures by the sensor to being fully processed and decent. On the GPU there are >20 stages to the ISP. The CSI-2 port where the camera attaches is just the first stage. That produces a raw bayer image, which is bebayered, black level corrected, lens shading corrected, denoised, sharpened, colour balanced, runs through the gain control, the white balance control. And that's just a subset of the processing...all done on the GPU at 30fps for 1080p.

Note that some camera sensors have all the processing built in - in fact the one being used does, but we don't actually use the internal ISP in this case, we use the one on the GPU, and just take bayer data from the camera.
It still sounds like many of the processes mentioned above could be done by cleverly written fragment shaders (render-to-texture) if public interfaces to use GLSL or Cg exist.
jamesh wrote:ref: You may be right should you 'you are right'
<<<Attack packet dropped by firewall>>>
They probably could be done with fragment shaders, but would be slower (general purpose code vs dedicated HW - code usually loses). Would take ages to write them all as well, and would''t provide the same level of programmability as available to the tuning on the GPU. The GPU support OGLES2.0, so you could do some GPGPU stuff if you were so inclined.

It's going to come down to speed. Rather than a fully GPU pipeline, you will be transferring the (large) bayer chunk of data to the Arm where it would be processed by a 700Mhz processor (compared with the 16way Vector code at 250Mhz = 4000Mhz , of which there are two, but not used exclusively for the ISP) with some bits passed back to the GPU for shader processing with the associated overheads. Lots of work for something that is slower than what is there already. Only benefit is user control of the whole tuning process.

In fact, as ARM's get faster there is benefit to running parts of the ISP on them. Not on this chip though.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

GauVeldt
Posts: 24
Joined: Thu Mar 14, 2013 4:46 pm
Contact: Website

Re: Camera Interface Specs

Sat Mar 16, 2013 8:57 am

jamesh wrote:you will be transferring the (large) bayer chunk of data to the Arm where it would be processed by a 700Mhz processor
Don't shaders need to work on textures in VRAM (ie: GPU's allocation)?
In which case leave the bayer in VRAM until the shaders have done whatever conversions (hopefully the debayer process can be coded as a GLSL/Cg shader depending on the fragment shader model that is supported). Don't move it to the ARM side until processing is required that cannot be done via shaders and has no public interface to perform via GPU directly.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 25393
Joined: Sat Jul 30, 2011 7:41 pm

Re: Camera Interface Specs

Sat Mar 16, 2013 11:33 am

GauVeldt wrote:
jamesh wrote:you will be transferring the (large) bayer chunk of data to the Arm where it would be processed by a 700Mhz processor
Don't shaders need to work on textures in VRAM (ie: GPU's allocation)?
In which case leave the bayer in VRAM until the shaders have done whatever conversions (hopefully the debayer process can be coded as a GLSL/Cg shader depending on the fragment shader model that is supported). Don't move it to the ARM side until processing is required that cannot be done via shaders and has no public interface to perform via GPU directly.
I suppose that might be possible with a huge amount of Broadcom software effort. Which means it'll never happen since we already have something that works fine - all this does is contrive a convoluted route for data though the system to replace a nice easy route for data through the system. ie. replacing the dedicated devices specifically built for the purpose with something that isn't. And for what?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

Hardware_man
Posts: 94
Joined: Tue Dec 04, 2012 6:28 pm
Contact: Website

Re: Camera Interface Specs

Sat Mar 16, 2013 3:58 pm

When I said, “Don’t blame the 1%, blame the bizarre” this is what I meant. I’ll bet the Foundation gets requests for new firmware every day. They couldn’t possibly write firmware for each request. So they have to prioritize. If maybe 100 people (the “bizarre”) all added to this thread and all said a general purpose video input would be a great thing and maybe gave example applications, that would have increased the priority. But actually about 5 people joined this thread and said they also wanted a general purpose video input. If I were the Foundation, I wouldn’t spend a lot of resources writing firmware because 5 people requested it.

If you wanted to do all of your video processing in the Arm, there are faster and multi core Arm chips available. The “magic” of the Broadcom chip is the dedicated video processing hardware.

Hardware_man

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 25393
Joined: Sat Jul 30, 2011 7:41 pm

Re: Camera Interface Specs

Sat Mar 16, 2013 4:07 pm

Hardware_man wrote:When I said, “Don’t blame the 1%, blame the bizarre” this is what I meant. I’ll bet the Foundation gets requests for new firmware every day. They couldn’t possibly write firmware for each request. So they have to prioritize. If maybe 100 people (the “bizarre”) all added to this thread and all said a general purpose video input would be a great thing and maybe gave example applications, that would have increased the priority. But actually about 5 people joined this thread and said they also wanted a general purpose video input. If I were the Foundation, I wouldn’t spend a lot of resources writing firmware because 5 people requested it.

If you wanted to do all of your video processing in the Arm, there are faster and multi core Arm chips available. The “magic” of the Broadcom chip is the dedicated video processing hardware.

Hardware_man
Spot on, although even a hundred people would be too few for a complicated feature like HDMI in or the proposal above. That said, I'm pretty sure >100 people would want HDMI in (I'd say thousands). Not so sure about the convoluted mechanism for ISP replacement on the ARM though.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

GauVeldt
Posts: 24
Joined: Thu Mar 14, 2013 4:46 pm
Contact: Website

Re: Camera Interface Specs

Sun Mar 17, 2013 3:15 am

jamesh wrote:
GauVeldt wrote:
jamesh wrote:you will be transferring the (large) bayer chunk of data to the Arm where it would be processed by a 700Mhz processor
Don't shaders need to work on textures in VRAM (ie: GPU's allocation)?
In which case leave the bayer in VRAM until the shaders have done whatever conversions (hopefully the debayer process can be coded as a GLSL/Cg shader depending on the fragment shader model that is supported). Don't move it to the ARM side until processing is required that cannot be done via shaders and has no public interface to perform via GPU directly.
I suppose that might be possible with a huge amount of Broadcom software effort. Which means it'll never happen since we already have something that works fine - all this does is contrive a convoluted route for data though the system to replace a nice easy route for data through the system. ie. replacing the dedicated devices specifically built for the purpose with something that isn't. And for what?
Guess the only option for general video input is to try and squish the USB bugs so a USB capture device could be used.

I see that CSI2 for this purpose is thoroughly stonewalled.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 25393
Joined: Sat Jul 30, 2011 7:41 pm

Re: Camera Interface Specs

Sun Mar 17, 2013 8:54 am

GauVeldt wrote:
jamesh wrote:
GauVeldt wrote: Don't shaders need to work on textures in VRAM (ie: GPU's allocation)?
In which case leave the bayer in VRAM until the shaders have done whatever conversions (hopefully the debayer process can be coded as a GLSL/Cg shader depending on the fragment shader model that is supported). Don't move it to the ARM side until processing is required that cannot be done via shaders and has no public interface to perform via GPU directly.
I suppose that might be possible with a huge amount of Broadcom software effort. Which means it'll never happen since we already have something that works fine - all this does is contrive a convoluted route for data though the system to replace a nice easy route for data through the system. ie. replacing the dedicated devices specifically built for the purpose with something that isn't. And for what?
Guess the only option for general video input is to try and squish the USB bugs so a USB capture device could be used.

I see that CSI2 for this purpose is thoroughly stonewalled.
For the moment, yes, USB is the only option. There have been some recent bug fixes that may help with the video side of USB.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

gordon77
Posts: 4630
Joined: Sun Aug 05, 2012 3:12 pm

Re: Camera Interface Specs

Sun Mar 17, 2013 9:00 am

When you say video side of USB do you mean specifically for this camera or fixes for all USB devices eg webcams ?

Gordon77

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 25393
Joined: Sat Jul 30, 2011 7:41 pm

Re: Camera Interface Specs

Sun Mar 17, 2013 9:17 am

gordon77 wrote:When you say video side of USB do you mean specifically for this camera or fixes for all USB devices eg webcams ?

Gordon77
Well, there a number of issues, and we are gradually knocking the list down. See the USB redux thread an the USB serial devices thread.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

itimpi
Posts: 1090
Joined: Sun Sep 25, 2011 11:44 am
Location: Potters Bar, United Kingdom
Contact: Website

Re: Camera Interface Specs

Sun Mar 17, 2013 10:24 am

gordon77 wrote:When you say video side of USB do you mean specifically for this camera or fixes for all USB devices eg webcams ?

Gordon77
I assume it means all USB cameras as one of the points about the Foundation's camera is that it does NOT connect via USB (and thus suffer from any limitations of the USB implementation).

Hardware_man
Posts: 94
Joined: Tue Dec 04, 2012 6:28 pm
Contact: Website

Re: Camera Interface Specs

Sun Mar 17, 2013 6:29 pm

If you look a few pages back in this thread, I have discussed using USB as a camera input for non compressed video. 720P (720 x 1280) at 29.97 FPS, 16 bits per pixel Y Cb Cr is right around the theoretical USB2.0 maximum data rate. JamesH pointed out that USB has a lot of overhead so you probably won’t get near the theoretical maximum data rate. And if I remember correctly, inside the Broadcom chip, the USB is connected directly to the ARM, not to the GPU. So you end up with the problem of having to move a lot of data, real fast from the ARM to the GPU. If I understand correctly, there is no DMA function to do this, just LOTS of MOV instructions.

The “other direction” that is, Foundation camera to CSI-2. Then you can compress the data with the H.264 or MPEG2 and then move the data to the USB as an output (like a web camera application). Now the data is already compressed before you have to move it from the GPU to the ARM. So you don’t have the volume of data in this direction.

JamesH, please correct if I don’t have this exactly right.

Hardware_man

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1489
Joined: Sat Sep 10, 2011 11:43 am

Re: Camera Interface Specs

Sun Mar 17, 2013 7:34 pm

Pretty much exactly... The CSI2 camera interface results in a number of different formats (you can access the raw bayer if you want to, but won't be able to do anything with it in real time) but yes the image comes out of the camera does about ten different image processing functions (in hardware) a couple (in software) and is then tunnelled into the H264 encoder. From here you get a stream on the ARM that you could transmit somewhere or save onto the SDCard or USB harddrive.


Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

GekkePrutser
Posts: 36
Joined: Sat Mar 23, 2013 5:09 pm

Re: Camera Interface Specs

Sat Mar 23, 2013 5:10 pm

Would there be a datasheet for the camera somewhere? I mean for the whole module as sold by the foundation, not just the Camera.

The reason is that I'd like to design a casing for it to have 3D printed and a datasheet with the exact dimensions would be very helpful.

User avatar
jbeale
Posts: 3581
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: Camera Interface Specs

Mon Mar 25, 2013 8:04 pm

There's a post somewhere calling out 25 mm along the long dimension (as I vaguely recall) but I have not seen a "data sheet" as such. At least one beta tester (Michael Horne, aka recantha) already has a camera board, marked "Rev 1.0", see: http://www.recantha.co.uk/blog/?p=2937 so you could ask him if he could do some measurements on it.

Hardware_man
Posts: 94
Joined: Tue Dec 04, 2012 6:28 pm
Contact: Website

Re: Camera Interface Specs

Tue Mar 26, 2013 1:14 pm

Does OpenELEC add support for non compressed video input to the Pi?

Hardware_man

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 25393
Joined: Sat Jul 30, 2011 7:41 pm

Re: Camera Interface Specs

Tue Mar 26, 2013 2:23 pm

Hardware_man wrote:Does OpenELEC add support for non compressed video input to the Pi?

Hardware_man
Not sure how it could without extra HW and code on the GPU.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

Ashrael
Posts: 1
Joined: Tue Mar 26, 2013 9:07 pm

Re: Camera Interface Specs

Tue Mar 26, 2013 9:21 pm

Hi all,

my first post :)

I got my pi last Sunday and I want to use it for my home security system.
I want to attach a camera to it, but at quite some distance. The CSI interface can only be used at short distance. Is there a way to create an extra network device (or more) on the Pi? That would solve my problem. I do not want to use wireless 'security' cameras :lol:. If there is a possibility to connect a composite camera (or more) directly, or through an interface, to the Pi that would also do.

Thanks!

ghans
Posts: 7882
Joined: Mon Dec 12, 2011 8:30 pm
Location: Germany

Re: Camera Interface Specs

Tue Mar 26, 2013 10:36 pm

What is the problem with wireless IP cams ?
You can only connect one camera module to the Pi.
Some very lucky people have got several USB cams
working simultaneously . That said , if you can get
the video into the Pi , sending it over the network
is pretty easy. It has been demoed with the camera module
and for Webcams you can use one of the hundreds of
Linux solutions.


ghans
• Don't like the board ? Missing features ? Change to the prosilver theme ! You can find it in your settings.
• Don't like to search the forum BEFORE posting 'cos it's useless ? Try googling : yoursearchtermshere site:raspberrypi.org

User avatar
recantha2
Posts: 290
Joined: Wed Nov 14, 2012 9:34 am
Location: Potton, Bedfordshire
Contact: Website Facebook Twitter

Re: Camera Interface Specs

Wed Mar 27, 2013 7:00 am

This is going to sound very glib, but how about a *really long* USB cable?
--
Michael Horne - @recantha
Raspberry Pi blog - http://www.recantha.co.uk/blog

Cambridge Raspberry Jam
Website: http://camjam.me
Facebook: https://www.facebook.com/cambridgeraspberryjam
Follow the Cambridge Raspberry Jam on Twitter - @cambridgejam

Hardware_man
Posts: 94
Joined: Tue Dec 04, 2012 6:28 pm
Contact: Website

Re: Camera Interface Specs

Wed Mar 27, 2013 4:22 pm

The USB specification has limits on cable lenght. I don't remember off the top of my head what it is, but you can easily find the USB 2.0 specification on line.

Ethernet is designed to work with very long cable lengths. The Pi board is small. For security camera applications, put a Foundation camera and Pi board at each camera location. Then you can use the Pi board to compress the video out of the camera and transmit the video over Ethernet CAT5 cable.

Hardware_man

rasmus25
Posts: 13
Joined: Mon Feb 04, 2013 3:53 pm

Re: Camera Interface Specs

Thu Mar 28, 2013 6:41 pm

jamesh wrote:
GauVeldt wrote:
jamesh wrote:you will be transferring the (large) bayer chunk of data to the Arm where it would be processed by a 700Mhz processor
Don't shaders need to work on textures in VRAM (ie: GPU's allocation)?
In which case leave the bayer in VRAM until the shaders have done whatever conversions (hopefully the debayer process can be coded as a GLSL/Cg shader depending on the fragment shader model that is supported). Don't move it to the ARM side until processing is required that cannot be done via shaders and has no public interface to perform via GPU directly.
I suppose that might be possible with a huge amount of Broadcom software effort. Which means it'll never happen since we already have something that works fine - all this does is contrive a convoluted route for data though the system to replace a nice easy route for data through the system. ie. replacing the dedicated devices specifically built for the purpose with something that isn't. And for what?
I managed to de-bayer my camera's image in GPU using these shaders: http://graphics.cs.williams.edu/papers/BayerJGT09/ (with very minor modifications). So this is not something that we have to wait for Broadcom to implement. Right now I am trying to figure out how to send this data directly to h.264 encoder, without copying it to CPU-s memory and back again. Copying a 1280x960 RGB buffer from VRAM to RAM takes about 40 ms.

Hardware_man
Posts: 94
Joined: Tue Dec 04, 2012 6:28 pm
Contact: Website

Re: Camera Interface Specs

Sat Apr 06, 2013 3:13 pm

JamesH,

Due to current geo-political issues, it is possible that Samsung might face delays getting their latest smart phones to market. If this is true, and if this relaxes some of Broadcom’s time frames, this might be a good time to get a universal video input driver written.

Hardware_man

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 25393
Joined: Sat Jul 30, 2011 7:41 pm

Re: Camera Interface Specs

Sat Apr 06, 2013 6:37 pm

Hardware_man wrote:JamesH,

Due to current geo-political issues, it is possible that Samsung might face delays getting their latest smart phones to market. If this is true, and if this relaxes some of Broadcom’s time frames, this might be a good time to get a universal video input driver written.

Hardware_man
Not my call I'm afraid. Although I am sceptical that a 'universal video input' is even possible. HDMI in would be, but that wouldn't handle cameras. As I have said before, there was an attempt by Nokia to standardise the camera I/F (SMIA). It failed. Without a standard S/W interface, a universal camera driver would be very difficult/impossible.

That said, there are plans afoot to improve the driver system, and that may help somewhat when interfacing cameras. But not on the Raspi - and not in the near future - it's a major rewrite that will take man years of work.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

User avatar
jbeale
Posts: 3581
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: Camera Interface Specs

Sun Apr 07, 2013 12:31 am

jamesh wrote: Although I am sceptical that a 'universal video input' is even possible. HDMI in would be, but that wouldn't handle cameras. [...]
I know it's not on the agenda anytime soon, and maybe anytime ever. But many consumer video cameras, even consumer stills cameras with a video feature, now feature HDMI output, both live and in playback. I have two such myself (Panasonic, and Canon). So the ability to take HDMI in (unencrypted, obviously) would be quite a general-purpose thing and certainly open up a lot of possibilities.

Return to “Camera board”