6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7317
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Wed Feb 12, 2014 10:49 am

airsports wrote:How do I enable 'night' exposure mode?
As far as I can see, through v4l2-ctl I can only set auto_exposure to either 0 (auto) or 1 (manual). Am I missing something?
Currently you can't. The previous person writing this driver half linked it to V4L2_EXPOSURE_APERTURE_PRIORITY, but then made that setting inaccessible.
It seems the only way to support that within the V4L2 spec is to implement scene modes (V4L2_CID_SCENE_MODE under http://hverkuil.home.xs4all.nl/spec/med ... a-controls). Another one to add to the list of extensions needed.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7317
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Fri Feb 21, 2014 4:53 pm

OK, new updates incoming - sorry it's taken a few more days than planned.
I'm just pushing them internally now - please watch https://twitter.com/RPF_Dev_Updates for when it is actually out in the wild.
  • - Support for H264 profiles and levels.
    - Correct V4L2_PIX_FMT_BGR24 to being V4L2_PIX_FMT_RGB24. I did look at doing BGR888 support, but it was just getting too involved for the time available.
    - Adds support for the other flavours of YUYV, and also NV12 (Y plane, and then U/V interleaved plane). That was mainly because they were just there anyway.
    - Basic scene mode support. Currently just NIGHT and SPORTS at the moment, and only controlling exposure and metering modes. This can be extended later - it's just a table of the scene enum with the relevant settings at the top of controls.c. It's not had the amount of testing I would like to have given it, but people were wanting night exposure mode so I thought it was worth pushing it out there
    - exposure_dynamic_framerate (see below)
The one big one is fixing a bug in the sensor driver that wasn't clipping the exposure time correctly based on requested framerate. (This was the base reason behind James having issue with long exposures in RapsiStill as he was asking for 30fps but getting anything up to a second. The timeout was kicking in at 400ms based on supposed frame time of 33ms, so any exposure longer than that failed).
The downsides to this are that RaspiStill now needs to ask for a sensible framerate (0=variable and is what it should be asking for in stills mode). I have nudged JamesH to sort that :)

It does also affect V4L2 as on the back of it I've now implemented V4L2_CID_EXPOSURE_AUTO_PRIORITY / v4l2-ctl --set-ctrl=exposure_dynamic_framerate=[0|1] so that you can get a variable framerate based on exposure time. This should help those that are wanting to use the night scene mode to get longer exposures (select both scene-mode=8 and exposure_dynamic_framerate=1). I don't think it should actually change the behaviour of any apps already using V4L2, so hopefully it's just a new feature :)
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5331
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Official V4L2 driver

Fri Feb 21, 2014 9:52 pm

Firmware has been update. Use rpi-update to get it.

towolf
Posts: 421
Joined: Fri Jan 18, 2013 2:11 pm

Re: Official V4L2 driver

Sat Feb 22, 2014 5:00 pm

Thanks a lot, 6by9. Those are all very useful additions. The variable frame rate mode is highly appreciated. Of course in many video applications fixed fps is needed, but modern digital video often operates on timestamps and frame pts is stored in the container. In MP4, for example, the container frame rate is merely informational and every frame has a time stamp (unlike .avi).

One question regarding this. In exposure_dynamic_framerate=1 mode, what are the limits? I assume the lower limit is ~ 1 sec. What is the upper limit? Still 30 fps? So it will emit at most 30 frames per second but expo time would get shorter (down to 1/6000s)?

towolf
Posts: 421
Joined: Fri Jan 18, 2013 2:11 pm

Re: Official V4L2 driver

Sat Feb 22, 2014 5:11 pm

Another question that I have: I suspect that -- as scene brightness decreases -- the exposure code first ramps up the analog gain (ISO) and then ramps up exposure time (in "night" mode).

For an application that I have I would prefer it the other way around. Does that make sense?

Would it be possible to divulge what all the presets do? We have a lot of reports here on the forum where people tried to derive what the presets do, but nothing explicit.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23709
Joined: Sat Jul 30, 2011 7:41 pm

Re: Official V4L2 driver

Sat Feb 22, 2014 6:13 pm

IIRC, the analog gain is ramped, then the digital gain, but I have a feeling that exposure time will start to increase before the limit of digital x analog is reaches (the gain product). But Its the ratios of these different settings that varies over different exposure modes. So night will have a large gain product, and the curve will be different to sports mode for example.

There's no simple 'equation' that bolts it all together, just a load of interaction between curves based on gains, exposure times etc. It's why tuning cameras is such a time consuming job - getting all these curves to interact and do the right thing.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7317
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Mon Feb 24, 2014 12:09 pm

towolf wrote:Thanks a lot, 6by9. Those are all very useful additions. The variable frame rate mode is highly appreciated. Of course in many video applications fixed fps is needed, but modern digital video often operates on timestamps and frame pts is stored in the container. In MP4, for example, the container frame rate is merely informational and every frame has a time stamp (unlike .avi).
Indeed, any camcorder use case generally wants fixed frame rate, but most stills use cases want to improve image quality by allowing longer exposure times (generally limited by not wanting image blur on the preview). Where and how timestamps get attached to frames is always an entertaining question - I'm just glad we don't currently support B-frames otherwise you need to maintain both decode and presentation timestamps accurately.
We've got a funny one with Motion as it doesn't set the framerate with V4L2 and is therefore the exposure time is restricted by the default FPS of the driver. It seemed that exposure_dynamic_framerate may be a nicer solution to improving image quality than having to tweak the framerate.
towolf wrote:One question regarding this. In exposure_dynamic_framerate=1 mode, what are the limits? I assume the lower limit is ~ 1 sec. What is the upper limit? Still 30 fps? So it will emit at most 30 frames per second but expo time would get shorter (down to 1/6000s)?
I actually set the framerate limits as being a minimum of 1fps, and a maximum of whatever the defined framerate is, so

Code: Select all

v4l2-ctl -p 15
v4l2-ctl --set-ctrl=exposure_dynamic_framerate=1
would set a range of 1-15fps.
Yes, exposure can go down below the frame time of that max framerate if the light level warrants it.
towolf wrote:Another question that I have: I suspect that -- as scene brightness decreases -- the exposure code first ramps up the analog gain (ISO) and then ramps up exposure time (in "night" mode).

For an application that I have I would prefer it the other way around. Does that make sense?
I'm not involved in the AE algorithm directly, and James is right with his comment that it's involved.
It could be configured to do what you ask as it is set up with stages that have a max AG and exp time defined, however we can't really get into supporting everyone trying to define their own exposure modes.
I'm being cautious in what I'm saying here, but it may be possible to open the tuning up to tinkerers, however with no support provided (ie if your tuning settings don't work, then we won't be helping out with why not, and we won't be documenting it in detail either). Whether that can be got past the lawyers is another matter.
towolf wrote:Would it be possible to divulge what all the presets do? We have a lot of reports here on the forum where people tried to derive what the presets do, but nothing explicit.
I don't believe there is anything too sensitive in this area, so I'll give some general details for night mode here. It has four bands,
  • - 200usec-10msec, and up to x1.5 AG
    - 10ms-125ms, and up to x2.5 AG
    - 125ms-250ms, and up to x4.0 AG
    - 250ms-1s, and up to x8.0 AG.
It will allow a level of digital gain as well within each band, but I don't know enough detail.
Actually with those settings you will broadly get exposure time being increased in preference to AG, which is the situation that you wanted.

The default exposure mode maxes out at a 66ms exposure time to avoid too much motion blur on video, so those wanting better low light performance really will be wanting the night scene mode.
Sports is the only other mode that is currently exposed via V4L2, and that goes for short exposure times to really minimise motion blur. Max exp time in that mode is 16.6ms.

I hope that helps.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

shuckle
Posts: 565
Joined: Sun Aug 26, 2012 11:49 am
Location: Finland

Re: Official V4L2 driver

Wed Feb 26, 2014 7:57 am

I tested the new nightmode with dynamic framerate and the results are amazingly good. It can produce clear pictures in very low light now. It is better than my other test cameras (ps3 eye USB cam and FI8903W IP cam). But getting down to 1 fps is not really video anymore :D . Would it be easy/possible to limit the minimum fps to a certain configurable value? (And would that actually help or make the picture again too dark; need to test this more). My FI8903W:s go to 4 fps minimum for example. So instead of exposure_dynamic_framerate now being boolean, could it take a real minimum value like:
v4l2-ctl --set-ctrl=exposure_dynamic_framerate=4
to make it dynamic, but stop at 4 fps?

Anyway, this nightmode / exposure_dynamic_framerate seems to make it again possible to use raspicam as security camera. Thanks a lot for that!

Another feature, which I would see very usefull in security cameras is digital zoom. Is there any hope to get that feature into v4l2 driver?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7317
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Wed Feb 26, 2014 2:20 pm

shuckle wrote:I tested the new nightmode with dynamic framerate and the results are amazingly good. It can produce clear pictures in very low light now. It is better than my other test cameras (ps3 eye USB cam and FI8903W IP cam). But getting down to 1 fps is not really video anymore :D . Would it be easy/possible to limit the minimum fps to a certain configurable value? (And would that actually help or make the picture again too dark; need to test this more). My FI8903W:s go to 4 fps minimum for example. So instead of exposure_dynamic_framerate now being boolean, could it take a real minimum value like:
v4l2-ctl --set-ctrl=exposure_dynamic_framerate=4
to make it dynamic, but stop at 4 fps?
We're working within the V4L2 spec. If the spec had an option to make it something other than a boolean then we could have gone down that route, but it doesn't :( . The exposure mode really determines what the minimum fps/max exposure time is, although internally we do have a secondary parameter specifying the FPS.
There are too many use cases to keep everyone happy - whilst you want min 4fps, those doing astronomy photography will be complaining that they want the full 1s exposure and complaining they can't get more. When we have more time, we can look into a custom V4L2 ctrl to set the min framerate for exposure_dynamic_framerate, but for now you'd have to make a local mod to the V4L2 and build your own local driver.
shuckle wrote:Another feature, which I would see very usefull in security cameras is digital zoom. Is there any hope to get that feature into v4l2 driver?
It's still on the list. Digital zoom and cropping is done in a slightly awkward way in V4L2 which doesn't map directly onto our controls.
I'm working on removing the padding restrictions on raw image buffers first (currently testing patches), but it will be done when we get a chance (time for this driver is very restricted at the moment).
Alternatively if someone in the community fancied doing it then I'd be happy to advise - this is an open source V4L2 driver (even if it is talking to a closed source GPU) so we're not stopping you ;) RaspiStill should even give you most of the detail of the MMAL calls that you'd need....
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

kevin140473
Posts: 2
Joined: Sun Mar 02, 2014 4:20 pm

Re: Official V4L2 driver

Sun Mar 02, 2014 5:00 pm

Thanks for the update especially ISO, and scene modes

I have been testing the updated driver with my OpenCV code but can not seem to get the "exposure_time_absolute" to function in the same way as the older V4l2 driver. Previously setting exposure_time_absolute to 10000 resulted in exposure time of 1s (with Auto Exposure set to Manual Mode). Which was great for low light still image capture of a static scene.

With the updated driver changing "exposure_time_absolute" beyond 300-400 (hard to tell exactly) makes no difference. I have dynamic frame rate enabled, scene mode set to "None". It appears as if there is hard minimum frame rate of 30fps which prevents going beyond 33ms exposure time in manual mode.

Setting Scene mode to "Night" (which I assume places the camera in auto-exposure even if manual mode is enabled) the frame rate drops and I get a brighter image than possible with Manual Exposure Mode.

As a sanity check I have also tested with GUVCViewer and see the same (although GUVCViewer does not allow access to the ISO settings or exposure bias?)

I am guessing that dynamic frame rate is not getting enabled correctly when in scene mode=None and Manual Exposure is enabled. Toggling dynamic frame rate off when in Night Mode the image intensity drops and is similar to the maximum possible when in manual mode.

Quite possibly I am missing something, but it would be great to understand why manual exposure (I use it as part of a software based auto exposure algorithm) does not seem to function in quite the same way

Cheers.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23709
Joined: Sat Jul 30, 2011 7:41 pm

Re: Official V4L2 driver

Sun Mar 02, 2014 7:00 pm

kevin140473 wrote:Thanks for the update especially ISO, and scene modes

I have been testing the updated driver with my OpenCV code but can not seem to get the "exposure_time_absolute" to function in the same way as the older V4l2 driver. Previously setting exposure_time_absolute to 10000 resulted in exposure time of 1s (with Auto Exposure set to Manual Mode). Which was great for low light still image capture of a static scene.

With the updated driver changing "exposure_time_absolute" beyond 300-400 (hard to tell exactly) makes no difference. I have dynamic frame rate enabled, scene mode set to "None". It appears as if there is hard minimum frame rate of 30fps which prevents going beyond 33ms exposure time in manual mode.

Setting Scene mode to "Night" (which I assume places the camera in auto-exposure even if manual mode is enabled) the frame rate drops and I get a brighter image than possible with Manual Exposure Mode.

As a sanity check I have also tested with GUVCViewer and see the same (although GUVCViewer does not allow access to the ISO settings or exposure bias?)

I am guessing that dynamic frame rate is not getting enabled correctly when in scene mode=None and Manual Exposure is enabled. Toggling dynamic frame rate off when in Night Mode the image intensity drops and is similar to the maximum possible when in manual mode.

Quite possibly I am missing something, but it would be great to understand why manual exposure (I use it as part of a software based auto exposure algorithm) does not seem to function in quite the same way

Cheers.
Have you updated the firmware? There have been changes to the exposure code that may be required.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7317
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Mon Mar 03, 2014 10:36 am

kevin140473 wrote:Thanks for the update especially ISO, and scene modes

I have been testing the updated driver with my OpenCV code but can not seem to get the "exposure_time_absolute" to function in the same way as the older V4l2 driver. Previously setting exposure_time_absolute to 10000 resulted in exposure time of 1s (with Auto Exposure set to Manual Mode). Which was great for low light still image capture of a static scene.

With the updated driver changing "exposure_time_absolute" beyond 300-400 (hard to tell exactly) makes no difference. I have dynamic frame rate enabled, scene mode set to "None". It appears as if there is hard minimum frame rate of 30fps which prevents going beyond 33ms exposure time in manual mode.

Setting Scene mode to "Night" (which I assume places the camera in auto-exposure even if manual mode is enabled) the frame rate drops and I get a brighter image than possible with Manual Exposure Mode.

As a sanity check I have also tested with GUVCViewer and see the same (although GUVCViewer does not allow access to the ISO settings or exposure bias?)

I am guessing that dynamic frame rate is not getting enabled correctly when in scene mode=None and Manual Exposure is enabled. Toggling dynamic frame rate off when in Night Mode the image intensity drops and is similar to the maximum possible when in manual mode.

Quite possibly I am missing something, but it would be great to understand why manual exposure (I use it as part of a software based auto exposure algorithm) does not seem to function in quite the same way

Cheers.
It's quite possible that I've made an error when adding scene modes/dynamic framerate. It gets tricky when you have 3 things all wanting to change the same value in different ways and I'll admit it didn't have the testing it really should have done.
I'll check today and report back (hopefully with a fix if I do find an issue).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7317
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Mon Mar 03, 2014 1:26 pm

6by9 wrote:I'll check today and report back (hopefully with a fix if I do find an issue).
http://hverkuil.home.xs4all.nl/spec/med ... a-controls section V4L2_CID_EXPOSURE_AUTO_PRIORITY
V4L2_CID_EXPOSURE_AUTO_PRIORITY boolean
When V4L2_CID_EXPOSURE_AUTO is set to AUTO or APERTURE_PRIORITY, this control determines if the device may dynamically vary the frame rate. By default this feature is disabled (0) and the frame rate must remain constant.
If you use V4L2_CID_EXPOSURE_AUTO(V4L2_EXPOSURE_MANUAL) / "v4l2-ctl --set-ctrl=auto_exposure=1" to request manual exposure, then the spec says we should abide by the frame rate requested, and so we do.

If you want manual exposure control to work in the way that you imply you do, then you'll also have to request a framerate that makes that exposure time possible, otherwise we will clip it. I've just done:
  • - v4l2-ctl --set-ctrl=auto_exposure=1
    - v4l2-ctl --set-ctrl=exposure_time_absolute=10000
    - v4l2-ctl -p 1
And I do get my 1 second exposure time. Without that last line, then it is limited at 33ms due to the framerate.

(It also explains why you got your old behaviour - the sensor driver had a bug that meant it didn't clip the exposure time requested, hence you got the time you requested to the detriment of the framerate. I fixed that bug in this release to deal with several other issues, and hence have changed your behaviour)
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

kevin140473
Posts: 2
Joined: Sun Mar 02, 2014 4:20 pm

Re: Official V4L2 driver

Mon Mar 03, 2014 9:19 pm

6by9 wrote:
6by9 wrote:I'll check today and report back (hopefully with a fix if I do find an issue).
http://hverkuil.home.xs4all.nl/spec/med ... a-controls section V4L2_CID_EXPOSURE_AUTO_PRIORITY
V4L2_CID_EXPOSURE_AUTO_PRIORITY boolean
When V4L2_CID_EXPOSURE_AUTO is set to AUTO or APERTURE_PRIORITY, this control determines if the device may dynamically vary the frame rate. By default this feature is disabled (0) and the frame rate must remain constant.
If you use V4L2_CID_EXPOSURE_AUTO(V4L2_EXPOSURE_MANUAL) / "v4l2-ctl --set-ctrl=auto_exposure=1" to request manual exposure, then the spec says we should abide by the frame rate requested, and so we do.

If you want manual exposure control to work in the way that you imply you do, then you'll also have to request a framerate that makes that exposure time possible, otherwise we will clip it. I've just done:
  • - v4l2-ctl --set-ctrl=auto_exposure=1
    - v4l2-ctl --set-ctrl=exposure_time_absolute=10000
    - v4l2-ctl -p 1
And I do get my 1 second exposure time. Without that last line, then it is limited at 33ms due to the framerate.

(It also explains why you got your old behaviour - the sensor driver had a bug that meant it didn't clip the exposure time requested, hence you got the time you requested to the detriment of the framerate. I fixed that bug in this release to deal with several other issues, and hence have changed your behaviour)
Brilliant that did the trick, thanks. Lack of understanding on my end - now I just have to work out how to set the max frame rate via the v4l2 header in python (as opposed to just calling the command line in python)

Thanks again

lucky2004alex
Posts: 1
Joined: Thu Jan 09, 2014 12:46 pm

Re: Official V4L2 driver

Thu Mar 06, 2014 3:06 pm

hello,
I have the same problem wit this driver.After i strat it (led on camera), all is freezing.ANy help?THX alex

magnatag
Posts: 33
Joined: Tue Mar 04, 2014 8:39 pm

Re: Official V4L2 driver

Thu Mar 06, 2014 5:45 pm

Is there anyway to use the new drivers to allow the RPiCamera to output only in grayscale to increase FPS? Mainly in using with OpenCV.

Thanks!

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23709
Joined: Sat Jul 30, 2011 7:41 pm

Re: Official V4L2 driver

Thu Mar 06, 2014 7:31 pm

Although greyscale won't really make the camera faster it would help in post processing. Can you use YUV format? That already has a greyscale/luma plane (Y)
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

magnatag
Posts: 33
Joined: Tue Mar 04, 2014 8:39 pm

Re: Official V4L2 driver

Thu Mar 06, 2014 8:29 pm

I can use YUV format! I just don't know how yet. Time to go digging in google!

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7317
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Fri Mar 07, 2014 2:59 pm

magnatag wrote:I can use YUV format! I just don't know how yet. Time to go digging in google!
- Select format V4L2_PIX_FMT_YUV420.
- Set a resolution <= 1280x720
- Start streaming.
- You'll get buffers back that are sized as (width * height * 1.5) bytes. Just look at from the start of the image and treat it as a width * height 8bits/pixel greyscale image and you're good. Ignore the extra (width * height * 0.5) bytes of data on the end - that's the chroma (colour) information that you're not wanting.

There would be almost no gain in switching from that format to a genuine grayscale image. We'd move a few fewer bytes around the place, but fairly insignificant in the scheme of things. You should be able to get 720P at 30fps through from the V4L2 driver - more than that and I'd be surprised if OpenCV can keep up on the ARM.
Until we get the 60fps sensor mode working (almost there!), then 30fps is the best you'll ever get out of the driver.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7317
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Fri Mar 07, 2014 3:05 pm

lucky2004alex wrote:hello,
I have the same problem wit this driver.After i strat it (led on camera), all is freezing.ANy help?THX alex
You haven't given us much information to go on.
What build are you using? The latest release should be fine, so have you done an rpi-update? And enabled the camera in raspi-config? Does raspistill work on the same Pi?
There aren't any known setup issues with the driver at the moment as released, so you'll have to do some investigation yourself.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

magnatag
Posts: 33
Joined: Tue Mar 04, 2014 8:39 pm

Re: Official V4L2 driver

Fri Mar 07, 2014 3:25 pm

Hi 6by9, thank you for the advice!

Would I select the format through v4l2-ctl commands? or set it inside opencv? I ran v4l2-ctl --list-formats and it does indeed show that my camera supports YU12 at index 0. The question is how do I specify that for my opencv stream to use that?

Is there a link with V4L2 commands?

Thanks!
6by9 wrote:
magnatag wrote:I can use YUV format! I just don't know how yet. Time to go digging in google!
- Select format V4L2_PIX_FMT_YUV420.
- Set a resolution <= 1280x720
- Start streaming.
- You'll get buffers back that are sized as (width * height * 1.5) bytes. Just look at from the start of the image and treat it as a width * height 8bits/pixel greyscale image and you're good. Ignore the extra (width * height * 0.5) bytes of data on the end - that's the chroma (colour) information that you're not wanting.

There would be almost no gain in switching from that format to a genuine grayscale image. We'd move a few fewer bytes around the place, but fairly insignificant in the scheme of things. You should be able to get 720P at 30fps through from the V4L2 driver - more than that and I'd be surprised if OpenCV can keep up on the ARM.
Until we get the 60fps sensor mode working (almost there!), then 30fps is the best you'll ever get out of the driver.

magnatag
Posts: 33
Joined: Tue Mar 04, 2014 8:39 pm

Re: Official V4L2 driver

Fri Mar 07, 2014 3:51 pm

So it looks like I can run this command:

Code: Select all

v4l2-ctl --set-fmt-video=width=1280,height=720,pixelformat=0
to set the size and format. Now the question is how do I stream and pick up the stream in opencv.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7317
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Fri Mar 07, 2014 4:28 pm

magnatag wrote:Hi 6by9, thank you for the advice!

Would I select the format through v4l2-ctl commands? or set it inside opencv? I ran v4l2-ctl --list-formats and it does indeed show that my camera supports YU12 at index 0. The question is how do I specify that for my opencv stream to use that?

Is there a link with V4L2 commands?

Thanks!
Sorry, but I have no idea how OpenCV integrates with V4L2. I would imagine that it would be setting up the format as otherwise it would have no idea how to interpret the data that it received.
...
A quick Google gave me http://docs.opencv.org/modules/highgui/ ... apture-set and I'd guess you want CV_CAP_PROP_FORMAT or CV_CAP_PROP_FOURCC.
http://answers.opencv.org/question/4346 ... fourcc-or/ implies that FOURCC doesn't work, and indeed that looks like the case. Even worse, having a quick read further through modules/highgui/src/cap_v4l.cpp from https://github.com/Itseez/opencv it looks like OpenCV is always converting to RGB888, so no matter what you ask V4L2 for, OpenCV will give you RGB888 :cry: And they also want to deal with YVU420, not YUV420, so it won't match up with our V4L2 driver either (if that were the only issue, then I could have a potential fix on the GPU to support YV12, aka YVU420, but it isn't).

I have no spare time to start playing with OpenCV, so I'm afraid you may be a little on your own here. I can't believe that OpenCV is really so limited, but perhaps people don't use it with V4L2 that heavily. The conversion to RGB888 can be done on the GPU but will drop the framerate slightly (that must be how it is currently working). If OpenCV can't cope with passing YUV/YVU420 directly out, or treating it as greyscale, then the conversion to RGB888 will really slow down on the ARM. Damned if you do, and damned if you don't.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7317
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Fri Mar 07, 2014 4:33 pm

magnatag wrote:So it looks like I can run this command:

Code: Select all

v4l2-ctl --set-fmt-video=width=1280,height=720,pixelformat=0
to set the size and format. Now the question is how do I stream and pick up the stream in opencv.
To confirm whether OpenCV really is setting the resolution and format, run your v4l2-ctl command before your app, and then

Code: Select all

v4l2-ctl -V
afterwards. It should print out the current mode that the V4L2 driver is in, and I'm suspecting that it'll be changed from your 1280x720 YU12 to some resolution and 'RGB3' as the pixel format.

Actually I may have made it worse for you with my latest release. We did claim to support BGR3, but we actually supported RGB3. I corrected the V4L2 driver to advertise RGB3, so OpenCV may now be dropping down the list of the formats that it supports and ending up at V4L2_PIX_FMT_YUYV. It goes from bad to worse!
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

magnatag
Posts: 33
Joined: Tue Mar 04, 2014 8:39 pm

Re: Official V4L2 driver

Fri Mar 07, 2014 5:59 pm

Oh boy! Well I managed to scale down my resolution down to 64x32 and surprisingly that will still give me enough data to work with. Not great, but workable for my application I guess.

You were right, I ran my program and then ran
  • v4l2-ctl --all
and I get the following output:

Code: Select all

Driver Info (not using libv4l2):
        Driver name   : bm2835 mmal
        Card type     : mmal service 15.1
        Bus info      : platform:bcm2835-v4l2
        Driver version: 3.10.32
        Capabilities  : 0x85000005
                Video Capture
                Video Overlay
                Read/Write
                Streaming
                Device Capabilities
        Device Caps   : 0x05000005
                Video Capture
                Video Overlay
                Read/Write
                Streaming
Priority: 2
Video input : 0 (Camera 0: ok)
Format Video Capture:
        Width/Height  : 64/32
        Pixel Format  : 'RGB3'
        Field         : None
        Bytes per Line: 192
        Size Image    : 6144
        Colorspace    : SRGB
Format Video Overlay:
        Left/Top    : 150/50
        Width/Height: 1024/768
        Field       : None
        Chroma Key  : 0x00000000
        Global Alpha: 0x00
        Clip Count  : 0
        Clip Bitmap : No
Framebuffer Format:
        Capability    :
        Flags         : Overlay Matches Capture/Output Size
        Width         : 64
        Height        : 32
        Pixel Format  : 'YU12'
        Bytes per Line: 96
        Size image    : 3072
        Colorspace    : JPEG (JFIF/ITU601)
Streaming Parameters Video Capture:
        Capabilities     : timeperframe
        Frames per second: 30.000 (30/1)
        Read buffers     : 1

User Controls

                     brightness (int)    : min=0 max=100 step=1 default=50 value              =50 flags=slider
                       contrast (int)    : min=-100 max=100 step=1 default=0 val              ue=0 flags=slider
                     saturation (int)    : min=-100 max=100 step=1 default=0 val              ue=0 flags=slider
                horizontal_flip (bool)   : default=0 value=0
                  vertical_flip (bool)   : default=0 value=0
           power_line_frequency (menu)   : min=0 max=3 default=1 value=1
                      sharpness (int)    : min=-100 max=100 step=1 default=0 val              ue=0 flags=slider
                  color_effects (menu)   : min=0 max=15 default=0 value=0
                         rotate (int)    : min=0 max=360 step=90 default=0 value              =0
             color_effects_cbcr (int)    : min=0 max=65535 step=1 default=32896               value=32896

MPEG Encoder Controls

             video_bitrate_mode (menu)   : min=0 max=1 default=0 value=0 flags=u              pdate
                  video_bitrate (int)    : min=25000 max=25000000 step=25000 def              ault=10000000 value=10000000
         repeat_sequence_header (bool)   : default=0 value=0
                     h264_level (menu)   : min=0 max=11 default=11 value=11
                   h264_profile (menu)   : min=0 max=4 default=4 value=4

Camera Controls

                  auto_exposure (menu)   : min=0 max=3 default=0 value=0
         exposure_time_absolute (int)    : min=1 max=10000 step=1 default=1000 v              alue=1000
     exposure_dynamic_framerate (bool)   : default=0 value=0
             auto_exposure_bias (intmenu): min=0 max=24 default=12 value=12
      white_balance_auto_preset (menu)   : min=0 max=9 default=1 value=1
            image_stabilization (bool)   : default=0 value=0
                iso_sensitivity (intmenu): min=0 max=4 default=0 value=0
         exposure_metering_mode (menu)   : min=0 max=2 default=0 value=0
                     scene_mode (menu)   : min=0 max=13 default=0 value=0

JPEG Compression Controls

            compression_quality (int)    : min=1 max=100 step=1 default=30 value              =30
Couple of thoughts here. Why is it saying "Driver Info (not using libv4l2)"? It's also interesting to see that OpenCv does change all the parameters just like you said (64/32 and RGB3) but Framebuffer Format is YU12.

I've also compiled opencv with this patch http://code.opencv.org/issues/3554 to allow me to use v4l2, thanks to bdsz!

It's a real bummer that I can't access YUV mode directly to speed up the whole system.

Return to “Camera board”