image goes black after a couple of minutes

[flok]

Hi,

I've connected a IR camera (the regular rpi camera board sold without the ir filter) to a model b (the one with 512MB ram and ethernet). I've installed the latest kernel (3.10.27+) with the video4linux2 driver.
When I run "koudevoeten" (http://www.vanheusden.com/koudevoeten/), things work fine but then after a couple of minutes of being idle (see below), the image becomes almost(!) black (almost: I vaguely see the leds of the RPI itself). Closing and reopening the device solves this (e.g. restarting the application).
A logical first reaction would be: koudevoeten is broken. I don't think that is the case as it all works fine on other Linux systems. I have for example a webcam at my frontdoor running for months (it could've been longer without power drops) without any hickups (asus eee laptop with external usb webcam).
What the program does is:
- open the video device
- configure it (e.g. resolution (1920x1080), pixel encoding (YUYV), streaming capture, frame retrieval via mmap)
- loop forever executing VIDIOC_DQBUF, copying the data (if there are any clients connected) and then invoking VIDIOC_QBUF
So even if it is idle, it will execute VIDIOC_(D)QBUF, it will just not memcpy the mmap contents elsewhere.

Any ideas?

[jamesh]

If you run raspistill do you see the same problem? For example in timelapse mode?

Sounds like the exposure is going wrong. Which makes me think something is starting the camera, not waiting long enough for the exposure to settle, then taking the picture, then closing. You need about 0.5s for the exposure to settle afte starting the camera, then take the picture.

[flok]

This does not happen with raspistill in timelapse mode.

The "koudevoeten" application opens the device once and keeps it open: it constantly loops executing VIDIOC_DQBUF, copying the data and executing VIDIOC_QBUF (e.g. regular video4linux2 frame-grabbing ioctls).

[jamesh]

Does it run a preview frame? If not then I think the exposure may slowly go out of kilter - the exposure calculation is generated from preview frames.

Might be a problem with the V4L driver not generating previews in the background. You are using the Raspi foundation driver?

[flok]

Hi,

No preview is shown: the program retrieves the yuyv data directly from the camera using the video4linux2 driver included in the kernel which I think is indeed from the raspberry pi foundation.

[jamesh]

I've asked the guy who wrote some of the V4L driver to take a look.

[flok]

Ok. De code is open source by the way: http://www.vanheusden.com/koudevoeten/ (MJPEG, MPNG, still jpg/png and VNC).

Code: Select all

koudevoeten -d /dev/video0 -p 8080 -W 1920 -H 1280 -t lower-right -q 50

and then point your browser to http://localhost:8080/image.jpg

[6by9]

1920x1080 YUYV will be going through the stills capture path - that'll limit the max capture rate apart from anything else.

I can't remember the full detail of how the exposure calcs run during stills capture. James is correct that most of the adjustment happens when preview is running, so multiple stills captures (JPEG, or YUV/YUYV/RGB resolution over 720P) may hit issues.

Reading the main text from http://www.vanheusden.com/koudevoeten/

Koudevoeten is a small application which streams (MJPEG stream) the video stream of any video4linux device via it's build-in webserver.

MJPEG streams can be viewed with most webbrowsers (apart from internet explorer, of course).

If it's asking for YUYV then it must be doing the JPEG/MJPEG encode on the ARM. The V4L2 device does offer hardware accelerated JPEG and MJPEG streaming. Is there a reason why you're choosing not to use it?
Even more bizarre, the code appears to be doing a conversion from the YUV420/YUYV that is captured to RGB. That means the JPEG library will have to be converting back to YUV again to encode the image.

I think my recommendation here is to either do something a little smarter in your app to ask for MJPEG, or to reduce the resolution to 720P so that it goes through the video capture path. I will make a note to have a look at exposure issues on multiple still captures, but I haven't got time to look into it now - sorry.

[flok]

1920x1080 YUYV will be going through the stills capture path - that'll limit the max capture rate apart from anything else.
I can't remember the full detail of how the exposure calcs run during stills capture. James is correct that most of the adjustment happens when preview is running, so multiple stills captures (JPEG, or YUV/YUYV/RGB resolution over 720P) may hit issues.

Any ways around that?

Reading the main text from http://www.vanheusden.com/koudevoeten/

Koudevoeten is a small application which streams (MJPEG stream) the video stream of any video4linux device via it's build-in webserver. MJPEG streams can be viewed with most webbrowsers (apart from internet explorer, of course).

If it's asking for YUYV then it must be doing the JPEG/MJPEG encode on the ARM. The V4L2 device does offer hardware accelerated JPEG and MJPEG streaming. Is there a reason why you're choosing not to use it?

Koudevoeten is originally a generic video4linux capturing application. It is only since yesterday that I tried getting it to work on the Raspberry Pi.

Even more bizarre, the code appears to be doing a conversion from the YUV420/YUYV that is captured to RGB. That means the JPEG library will have to be converting back to YUV again to encode the image.

Yes, that's an optimalization I'm pondering about.
The reason it was done this way is to have a generic pixel format available to "consumers" of the source-object: that way I don't need multiple implementations for e.g. VNC support.

I think my recommendation here is to either do something a little smarter in your app to ask for MJPEG, or to reduce the resolution to 720P so that it goes through the video capture path. I will make a note to have a look at exposure issues on multiple still captures, but I haven't got time to look into it now - sorry.

Ok I'll add a note to the website regarding the 720P work-around.
Please note that this problem is probably not specific for koudevoeten: other video4linux applications will hit this problem as well (unless they use the 720P resolution as you suggested).
[...]
I verified that when using 1280x720 there's no problem with exposure.
Is the exposure problem something I can override/fix with a v4l2 call?
Especially since when using 1280x720 not the whole image is available: part of it is cut-off(!).

regards

[6by9]

I think my recommendation here is to either do something a little smarter in your app to ask for MJPEG, or to reduce the resolution to 720P so that it goes through the video capture path. I will make a note to have a look at exposure issues on multiple still captures, but I haven't got time to look into it now - sorry.

Ok I'll add a note to the website regarding the 720P work-around.
Please note that this problem is probably not specific for koudevoeten: other video4linux applications will hit this problem as well (unless they use the 720P resolution as you suggested).
[...]
I verified that when using 1280x720 there's no problem with exposure.
Is the exposure problem something I can override/fix with a v4l2 call?
Especially since when using 1280x720 not the whole image is available: part of it is cut-off(!).

regards

No, there's no way to fix this via V4L2 except possibly running the overlay at the same time as your captures.

The field of view difference is being looked at, but the settings provided by the sensor manufacturer just aren't working at the moment.
The issue is that the sensor can not stream the full image at 30fps, so only reads out the central 1920x1080 pixels from the middle of the 2592x1944 array. The new settings are meant to subsample both horizontally and vertically on the sensor to read out the whole frame as 1296x972. The downside is that this is then only just greater than 720P, so 1080P recording will always need to be cropped with this sensor.

[jamesh]

I think its probably worth using the binned mode for 1080p as well, and just scaling it up a little at the end. Doubt there will be much difference in image quality - less than 100 pixels vertically.

[flok]

I think its probably worth using the binned mode for 1080p as well, and just scaling it up a little at the end. Doubt there will be much difference in image quality - less than 100 pixels vertically.

It is not neccessarily quality I miss, but parts of the room I try to monitor which is unfortunate.

[jamesh]

Yes, it's a known problem, and we are looking at it. Omnivision have got back, so hopefully some movement on this soon.

[flok]

thank you

[towolf]

I think its probably worth using the binned mode for 1080p as well, and just scaling it up a little at the end. Doubt there will be much difference in image quality - less than 100 pixels vertically.

If you do that please make it explicit what is happening behind the scenes.

If you can generate roundabout 720p from binned output then what we get is 720p. Scaling that up and calling it 1080p is kinda akin to deceptive packaging.

BTW, in Germany public TV in HD runs at 720p50, they opted for this instead of 1080i50 when they had to decide due to their bandwidth alotment, and I agree with that decision.

Personally, I'd prefer no scaling at all (besides the 2x2 binning).

[jamesh]

I think you will be surprised to know how many phone manufacturers use binned then scaled modes with no real loss of definition, and say it's 1080p. It's certainly slighter higher definition than 720p at 1296x972 compared to 1280x720.

But it is a tradeoff - do you want full FOV 1080p or not?

UK Freeview HD broadcast are also 720p, not sure of the frame rate.

[towolf]

I think you will be surprised to know how many phone manufacturers use binned then scaled modes with no real loss of definition, and say it's 1080p. It's certainly slighter higher definition than 720p at 1296x972 compared to 1280x720.

But it is a tradeoff - do you want full FOV 1080p or not?

Personally, I’m not insisting on 1080 for anything. I would love full FoV at 1296x972 px on output. And then I could, if needed, use a ROI or crop to 1280x720, I suppose.

[gordon77]

I think you will be surprised to know how many phone manufacturers use binned then scaled modes with no real loss of definition, and say it's 1080p. It's certainly slighter higher definition than 720p at 1296x972 compared to 1280x720.

But it is a tradeoff - do you want full FOV 1080p or not?

UK Freeview HD broadcast are also 720p, not sure of the frame rate.

Not all is 720p
http://www.bbc.co.uk/faqs/tv/bbchd_channels_schedule

And maybe most are 1080
http://www.avforums.com/threads/all-fre ... g.1530548/

[jamesh]

I think you will be surprised to know how many phone manufacturers use binned then scaled modes with no real loss of definition, and say it's 1080p. It's certainly slighter higher definition than 720p at 1296x972 compared to 1280x720.

But it is a tradeoff - do you want full FOV 1080p or not?

UK Freeview HD broadcast are also 720p, not sure of the frame rate.

Not all is 720p
http://www.bbc.co.uk/faqs/tv/bbchd_channels_schedule

And maybe most are 1080
http://www.avforums.com/threads/all-fre ... g.1530548/

Ah, interesting, I was under the impression it was all 720p.

[gordon77]

Maybe you need to check any settings on your TV

[jamesh]

That runs in 1080p always, the Freeview HD PVR (Humax) always scales up to 1080 what ever the input. So you really don't know what the source is unless you check.

[jbeale]

But it is a tradeoff - do you want full FOV 1080p or not?

I think if people could see still frame captures that show the difference (or lack of it) in this case, they might be less prone to argue just on counting pixels. I suspect there isn't much useful information to be gained by "true" 1080p vs scaled from this sensor, but that's just a suspicion. I also suspect in low-light you could get significantly more true image per-pixel information using binned+scaled, over cropped.

[flok]

Is capturing jpeg frames via video4linux known to be working?
Because I get only black frames in valid(!) jpeg format.
This is both in the 1280x720 as well as the 1920x1080 resolution.
The frames are immediately black, not after a while.
This is a console/terminal application without any preview windows.

[jamesh]

I think probably its not had enough time to expose the image correctly again. Try it with a preview and give it a second to gets its settings right.

[flok]

Hi James,

The device is constantly open, e.g. I constantly fetch frames from it so it should give an image if it was the issue you described.

Also: the device is used in a remote location (birdhouse but friends of mine would like to use it as a security cam) so I cannot run an X environment (for the preview) on it.