6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6424
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Tue Aug 07, 2018 2:39 pm

zim wrote:
Tue Aug 07, 2018 2:37 pm
Thank you for looking into it, and nice to know that there is a fix coming along!
No problem.
Thank you for the report, and sorry if I was initially dismissive. We do try our best to avoid regressions, but testing can never be totally exhaustive.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5213
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Official V4L2 driver

Wed Aug 08, 2018 5:19 pm

6by9 wrote:
Tue Aug 07, 2018 9:52 am
I don't know when the next rpi-update is going to be done, but the fix should be in there. I'll be doing some more testing in the meantime.
Fix should be present in latest rpi-update firmware.

judovana
Posts: 1
Joined: Tue Oct 23, 2018 4:04 pm

Re: Official V4L2 driver

Tue Oct 23, 2018 4:06 pm

Hi!

Is there a way I can build this driver for fedora? Or generally, was somebody able to make official cammera running on fedora? Is there better place to ask? (i'm running f29 on pi3 B+ aarch64)

dmatthews
Posts: 2
Joined: Thu Dec 06, 2018 3:18 am

Re: Official V4L2 driver

Thu Dec 06, 2018 3:41 am

Is it possible to access Bayer image data directly, without stuffing it into the EXIF data on a JPEG?

Earlier in this discussion, I found a passing mention of Bayer data getting transferred directly to memory before being read out again to the ISP. Is it possible to access the data at that point? Or to just pass the data through the GPU unchanged?

My application is pretty atypical, and I'm hoping to access full-sensor raw data faster than the 1-2 FPS I'm getting from the Python picamera module. If this is not possible, I would greatly appreciate it if someone could help me understand what's preventing it so I can stop banging my head against the wall.

Thanks for all your work!

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6424
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Thu Dec 06, 2018 3:02 pm

dmatthews wrote:
Thu Dec 06, 2018 3:41 am
Is it possible to access Bayer image data directly, without stuffing it into the EXIF data on a JPEG?

Earlier in this discussion, I found a passing mention of Bayer data getting transferred directly to memory before being read out again to the ISP. Is it possible to access the data at that point? Or to just pass the data through the GPU unchanged?

My application is pretty atypical, and I'm hoping to access full-sensor raw data faster than the 1-2 FPS I'm getting from the Python picamera module. If this is not possible, I would greatly appreciate it if someone could help me understand what's preventing it so I can stop banging my head against the wall.

Thanks for all your work!
Pretty much a duplicate of viewtopic.php?f=43&t=52203&p=1400929#p1400929

And there's the minor point that there is no way to get the Bayer data through the bcm2835-v4l2 driver (JPEG+RAW is not supported).

There is a very basic kernel driver for ov5647 (v1 camera) running in raw mode, but it only supports 640x480 at a fixed frame rate. That can be used with the bcm2835-unicam driver, but you're better off looking at raspiraw, as has already been referenced in the linked thread.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

hazeii
Posts: 30
Joined: Sat Apr 25, 2015 8:33 am

Re: Official V4L2 driver

Fri Apr 05, 2019 1:44 pm

Having recently switched to a new release of Raspbian, some custom software that reads stills from the camera via the V4L2 interface is going a lot slower. It appears to be because the camera is going into 'stills mode' at a lower resolution than before, despite the presence of 'max_video_height' and 'max_video_width' settings.

Here's how the module is being loaded:-

sudo modprobe bcm2835-v4l2 max_video_width=3280 max_video_height=2464

On an older system (4.4.13-v7+ #894), dmesg reports the following:-

Code: Select all

[   12.747648] media: Linux media interface: v0.10
[   12.764754] Linux video capture interface: v2.00
[   12.810751] bcm2835-v4l2: scene mode selected 0, was 0
[   12.811003] bcm2835-v4l2: V4L2 device registered as video0 - stills mode > 3280x2464 <--- Correct
On the new system (4.19.32-v7+ #1210, after an 'rpi-update' to see if that sorted the issue):

Code: Select all

[    4.203845] bcm2835-codec bcm2835-codec: Device registered as /dev/video10
[    4.203861] bcm2835-codec bcm2835-codec: Loaded V4L2 decode
[    4.208662] bcm2835-codec bcm2835-codec: Device registered as /dev/video11
[    4.208688] bcm2835-codec bcm2835-codec: Loaded V4L2 encode
[    4.227512] bcm2835-codec bcm2835-codec: Device registered as /dev/video12
[    4.227526] bcm2835-codec bcm2835-codec: Loaded V4L2 isp
[    4.239165] bcm2835-v4l2: scene mode selected 0, was 0
[    4.240636] bcm2835-v4l2: V4L2 device registered as video0 - stills mode > 1280x720 <--- Not as requested
[    4.250138] bcm2835-v4l2: Broadcom 2835 MMAL video capture ver 0.0.2 loaded.
Any suggestions on how to fix this, so we get 'video mode' (i.e. faster stills capture) as higher resolution, e.g. 1640x1232?

TIA!

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6424
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Fri Apr 05, 2019 2:30 pm

hazeii wrote:
Fri Apr 05, 2019 1:44 pm
Having recently switched to a new release of Raspbian, some custom software that reads stills from the camera via the V4L2 interface is going a lot slower. It appears to be because the camera is going into 'stills mode' at a lower resolution than before, despite the presence of 'max_video_height' and 'max_video_width' settings.

Here's how the module is being loaded:-

sudo modprobe bcm2835-v4l2 max_video_width=3280 max_video_height=2464

On an older system (4.4.13-v7+ #894), dmesg reports the following:-

Code: Select all

[   12.747648] media: Linux media interface: v0.10
[   12.764754] Linux video capture interface: v2.00
[   12.810751] bcm2835-v4l2: scene mode selected 0, was 0
[   12.811003] bcm2835-v4l2: V4L2 device registered as video0 - stills mode > 3280x2464 <--- Correct
On the new system (4.19.32-v7+ #1210, after an 'rpi-update' to see if that sorted the issue):

Code: Select all

[    4.203845] bcm2835-codec bcm2835-codec: Device registered as /dev/video10
[    4.203861] bcm2835-codec bcm2835-codec: Loaded V4L2 decode
[    4.208662] bcm2835-codec bcm2835-codec: Device registered as /dev/video11
[    4.208688] bcm2835-codec bcm2835-codec: Loaded V4L2 encode
[    4.227512] bcm2835-codec bcm2835-codec: Device registered as /dev/video12
[    4.227526] bcm2835-codec bcm2835-codec: Loaded V4L2 isp
[    4.239165] bcm2835-v4l2: scene mode selected 0, was 0
[    4.240636] bcm2835-v4l2: V4L2 device registered as video0 - stills mode > 1280x720 <--- Not as requested
[    4.250138] bcm2835-v4l2: Broadcom 2835 MMAL video capture ver 0.0.2 loaded.
Any suggestions on how to fix this, so we get 'video mode' (i.e. faster stills capture) as higher resolution, e.g. 1640x1232?
bcm2835-v4l2 is a platform driver now, so is automatically loaded at boot time. Your modprobe line is doing nothing.

Either unload bcm2835-v4l2 first, or create a file (name not important, but typically something like bcm2835_v4l2.conf) in /etc/modprobe.d/ containing

Code: Select all

options bcm2835-v4l2 max_video_width=3280
options bcm2835-v4l2 max_video_height=2464
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

hazeii
Posts: 30
Joined: Sat Apr 25, 2015 8:33 am

Re: Official V4L2 driver

Fri Apr 05, 2019 4:12 pm

Many thanks for the speedy reply, that solved it!

hazeii
Posts: 30
Joined: Sat Apr 25, 2015 8:33 am

Re: Official V4L2 driver

Mon Apr 08, 2019 9:29 am

Following on from the above, we're having problems grabbing big frames (inc. kernel crashes) with the latest firmware unless we increase GPU memory. There are varying symptoms depending on video format and frame size but possibly all related to some underlying memory issue. Easiest way I've found to reproduce is as follows:-

1. Stock install of 2018-11-13-raspbian-stretch-lite on Pi 3 B+ with Sony camera (with or without rpi-update)
2. Load video module with max_video_width/height set to 3280 and 2464
3. Build the capture.c example from v4l2.
4. Set video format: v4l2-ctl --set-fmt-video=width=3280,height=2464,pixelformat=YUV420
5. Run 'capture' example; result here is " VIDIOC_STREAMON error 1, Operation not permitted\n"' (whether root or not)
6. 'dmesg' reports error (see below).
7. Set video format: v4l2-ctl --set-fmt-video=width=640,height=480,pixelformat=YUV420
8. Run 'capture' again; generally results in system lockup.

Setting gpu_mem to 256 (from stock 128) and all works as expected, hence possible memory allocation issue? We're tight on memory so we don't want to have to give up the extra memory if it can be avoided.

This is on 4.19.32-v7+ #1210; the older version 4.4.13-v7+ #894 works as expected. Also reproducible by moving the SD card with the later version into older, known good systems (running the older version in the new systems also works as expected).

FWIW problems start showing up at 1640x1232, and YUYV seems to be more of an issue than YU12, e.g. setting max_video_height and width to 1920x1080 is ok for YU12 but hangs (without dmesg error) for YUYV. With these settings trying to capture at 3280x2484 (which would be in stills mode) works (slowly) for YU12 but fails for YUYV.

Here's the relevant bits from 'dmesg':

Code: Select all

[    5.247131] bcm2835-v4l2: scene mode selected 0, was 0
[    5.247548] bcm2835-v4l2: V4L2 device registered as video0 - stills mode > 3280x2464
[    5.253652] bcm2835-v4l2: Broadcom 2835 MMAL video capture ver 0.0.2 loaded.

Code: Select all

[  165.693646] bcm2835-v4l2: Failed to enable capture port - error -1. Disabling camera port again
[  165.720695] ------------[ cut here ]------------
[  165.720757] WARNING: CPU: 3 PID: 649 at drivers/media/common/videobuf2/videobuf2-core.c:1331 vb2_start_streaming+0xe4/0x160 [videobuf2_common]
[  165.720763] Modules linked in: cmac bnep hci_uart btbcm serdev bluetooth ecdh_generic evdev brcmfmac brcmutil sha256_generic cfg80211 bcm2835_codec(C) rfkill bcm2835_v4l2(C) v4l2_mem2mem v4l2_common bcm2835_mmal_vchiq(C) videobuf2_vmalloc videobuf2_dma_contig videobuf2_memops videobuf2_v4l2 snd_bcm2835(C) videobuf2_common raspberrypi_hwmon snd_pcm hwmon snd_timer videodev media snd vc_sm_cma(C) fixed uio_pdrv_genirq uio ip_tables x_tables ipv6
[  165.720981] CPU: 3 PID: 649 Comm: capture-v4l Tainted: G         C        4.19.32-v7+ #1210
[  165.720986] Hardware name: BCM2835
[  165.721020] [<80111e8c>] (unwind_backtrace) from [<8010d414>] (show_stack+0x20/0x24)
[  165.721038] [<8010d414>] (show_stack) from [<807ed5c0>] (dump_stack+0xd4/0x118)
[  165.721059] [<807ed5c0>] (dump_stack) from [<80120638>] (__warn+0x104/0x11c)
[  165.721075] [<80120638>] (__warn) from [<80120788>] (warn_slowpath_null+0x50/0x58)
[  165.721115] [<80120788>] (warn_slowpath_null) from [<7f1e96b0>] (vb2_start_streaming+0xe4/0x160 [videobuf2_common])
[  165.721188] [<7f1e96b0>] (vb2_start_streaming [videobuf2_common]) from [<7f1eb01c>] (vb2_core_streamon+0x130/0x16c [videobuf2_common])
[  165.721244] [<7f1eb01c>] (vb2_core_streamon [videobuf2_common]) from [<7f20f254>] (vb2_streamon+0x40/0x60 [videobuf2_v4l2])
[  165.721290] [<7f20f254>] (vb2_streamon [videobuf2_v4l2]) from [<7f20f2c4>] (vb2_ioctl_streamon+0x50/0x54 [videobuf2_v4l2])
[  165.721426] [<7f20f2c4>] (vb2_ioctl_streamon [videobuf2_v4l2]) from [<7f10fe3c>] (v4l_streamon+0x28/0x2c [videodev])
[  165.721623] [<7f10fe3c>] (v4l_streamon [videodev]) from [<7f1127a0>] (__video_do_ioctl+0x268/0x528 [videodev])
[  165.721821] [<7f1127a0>] (__video_do_ioctl [videodev]) from [<7f11643c>] (video_usercopy+0x218/0x600 [videodev])
[  165.722016] [<7f11643c>] (video_usercopy [videodev]) from [<7f116844>] (video_ioctl2+0x20/0x24 [videodev])
[  165.722209] [<7f116844>] (video_ioctl2 [videodev]) from [<7f10e144>] (v4l2_ioctl+0x4c/0x60 [videodev])
[  165.722320] [<7f10e144>] (v4l2_ioctl [videodev]) from [<802abc10>] (do_vfs_ioctl+0xbc/0x804)
[  165.722339] [<802abc10>] (do_vfs_ioctl) from [<802ac39c>] (ksys_ioctl+0x44/0x6c)
[  165.722355] [<802ac39c>] (ksys_ioctl) from [<802ac3dc>] (sys_ioctl+0x18/0x1c)
[  165.722371] [<802ac3dc>] (sys_ioctl) from [<80101000>] (ret_fast_syscall+0x0/0x28)
[  165.722379] Exception stack(0xb2985fa8 to 0xb2985ff0)
[  165.722392] 5fa0:                   0001202c 00000000 00000003 40045612 7ef5d650 40045612
[  165.722404] 5fc0: 0001202c 00000000 000107cc 00000036 00000000 00000000 76f6d000 7ef5d604
[  165.722413] 5fe0: 000226c8 7ef5d5e4 00010984 76eb0c7c
[  165.722422] ---[ end trace f315b2021c204c4e ]---

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6424
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Mon Apr 08, 2019 10:26 am

YUYV, being a YUV422 format, uses bigger buffers than YUV420 formats (16bpp vs 12bpp).

8MPix images are big!

Bayer raw 10 they're ~10MB each, and for video mode you need 3, or 30MB.
YUV422 frames are 16MPix each, and again video mode wants 3 for internal allocations, or 48MB.
Each buffer passed back to the ARM requires an equivalent sized buffer, so in the case of capture.c you are asking for 4 buffers in your request to VIDIOC_REQBUFS, so that's another 64MB if YUYV.

Total 142MB just in camera buffers, therefore it will fail if gpu_mem is 128M. You may get away with gpu_mem=160M. (You need to allow for an 8MB framebuffer, and some extra working memory for use cases).
Set gpu_mem to 256M and run your test case. "sudo vcdbg reloc" will tell you the amount of heap still free, so you can reduce gpu_mem by that amount.

It is very uncommon to use YUV422 for video mode in typical use cases, therefore I suspect the internal allocation isn't allowing for that. You may be able to fudge it by asking for "max_video_height=3286" (2464*16/12).

The jump from 4.4 to 4.19 is HUGE - that's well over 2 years worth of work. Please test on 4.9 and 4.14 if you want to try and track down vaguely with which step the requirements have changed. I don't recall any significant changes in that area.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

hazeii
Posts: 30
Joined: Sat Apr 25, 2015 8:33 am

Re: Official V4L2 driver

Mon Apr 08, 2019 12:44 pm

We're actually using YUV420 (for our purposes we only need Y), YUYV just made the issue easier to reproduce. Having done a bit of poking about with 'vcdbg reloc' now, it looks like we were just making it with GPU_MEM=128 on the old version (we're using 3 capture buffers rather than the 4 in the sample) and since the version appears to use a bit more memory it tipped it over the edge. That suggests we'll be able to get away with just a small increase in GPU_MEM.

Getting some impressive performance on the Pi 3 B+ now, streaming video 1280x720x8 video at 40FPS out via the LAN (~295Mbps), at higher res. obviously the network interface means rate/resolution tradeoffs (thinking about simple compression schemes to work around that).

Thanks for the explanation, detail and speedy reply!

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6424
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Mon Apr 08, 2019 2:18 pm

hazeii wrote:
Mon Apr 08, 2019 12:44 pm
We're actually using YUV420 (for our purposes we only need Y), YUYV just made the issue easier to reproduce. Having done a bit of poking about with 'vcdbg reloc' now, it looks like we were just making it with GPU_MEM=128 on the old version (we're using 3 capture buffers rather than the 4 in the sample) and since the version appears to use a bit more memory it tipped it over the edge. That suggests we'll be able to get away with just a small increase in GPU_MEM.

Getting some impressive performance on the Pi 3 B+ now, streaming video 1280x720x8 video at 40FPS out via the LAN (~295Mbps), at higher res. obviously the network interface means rate/resolution tradeoffs (thinking about simple compression schemes to work around that).

Thanks for the explanation, detail and speedy reply!
I'd need to check through exactly how the buffers are allocated. I suspect YUYV will almost always wedge at full resolution as the internal buffers aren't big enough.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Return to “Camera board”