Page 1 of 2

image goes black after a couple of minutes

Posted: Mon Jan 27, 2014 10:34 pm
by flok
Hi,

I've connected a IR camera (the regular rpi camera board sold without the ir filter) to a model b (the one with 512MB ram and ethernet). I've installed the latest kernel (3.10.27+) with the video4linux2 driver.
When I run "koudevoeten" (http://www.vanheusden.com/koudevoeten/), things work fine but then after a couple of minutes of being idle (see below), the image becomes almost(!) black (almost: I vaguely see the leds of the RPI itself). Closing and reopening the device solves this (e.g. restarting the application).
A logical first reaction would be: koudevoeten is broken. I don't think that is the case as it all works fine on other Linux systems. I have for example a webcam at my frontdoor running for months (it could've been longer without power drops) without any hickups (asus eee laptop with external usb webcam).
What the program does is:
- open the video device
- configure it (e.g. resolution (1920x1080), pixel encoding (YUYV), streaming capture, frame retrieval via mmap)
- loop forever executing VIDIOC_DQBUF, copying the data (if there are any clients connected) and then invoking VIDIOC_QBUF
So even if it is idle, it will execute VIDIOC_(D)QBUF, it will just not memcpy the mmap contents elsewhere.

Any ideas?

Re: image goes black after a couple of minutes

Posted: Mon Jan 27, 2014 10:51 pm
by jamesh
If you run raspistill do you see the same problem? For example in timelapse mode?

Sounds like the exposure is going wrong. Which makes me think something is starting the camera, not waiting long enough for the exposure to settle, then taking the picture, then closing. You need about 0.5s for the exposure to settle afte starting the camera, then take the picture.

Re: image goes black after a couple of minutes

Posted: Tue Jan 28, 2014 11:15 am
by flok
This does not happen with raspistill in timelapse mode.

The "koudevoeten" application opens the device once and keeps it open: it constantly loops executing VIDIOC_DQBUF, copying the data and executing VIDIOC_QBUF (e.g. regular video4linux2 frame-grabbing ioctls).

Re: image goes black after a couple of minutes

Posted: Tue Jan 28, 2014 11:25 am
by jamesh
Does it run a preview frame? If not then I think the exposure may slowly go out of kilter - the exposure calculation is generated from preview frames.

Might be a problem with the V4L driver not generating previews in the background. You are using the Raspi foundation driver?

Re: image goes black after a couple of minutes

Posted: Tue Jan 28, 2014 11:30 am
by flok
Hi,

No preview is shown: the program retrieves the yuyv data directly from the camera using the video4linux2 driver included in the kernel which I think is indeed from the raspberry pi foundation.

Re: image goes black after a couple of minutes

Posted: Tue Jan 28, 2014 11:43 am
by jamesh
I've asked the guy who wrote some of the V4L driver to take a look.

Re: image goes black after a couple of minutes

Posted: Tue Jan 28, 2014 11:54 am
by flok
Ok. De code is open source by the way: http://www.vanheusden.com/koudevoeten/ (MJPEG, MPNG, still jpg/png and VNC).

Code: Select all

koudevoeten -d /dev/video0 -p 8080 -W 1920 -H 1280 -t lower-right -q 50
and then point your browser to http://localhost:8080/image.jpg

Re: image goes black after a couple of minutes

Posted: Tue Jan 28, 2014 11:55 am
by 6by9
1920x1080 YUYV will be going through the stills capture path - that'll limit the max capture rate apart from anything else.

I can't remember the full detail of how the exposure calcs run during stills capture. James is correct that most of the adjustment happens when preview is running, so multiple stills captures (JPEG, or YUV/YUYV/RGB resolution over 720P) may hit issues.

Reading the main text from http://www.vanheusden.com/koudevoeten/
Koudevoeten is a small application which streams (MJPEG stream) the video stream of any video4linux device via it's build-in webserver.

MJPEG streams can be viewed with most webbrowsers (apart from internet explorer, of course).
If it's asking for YUYV then it must be doing the JPEG/MJPEG encode on the ARM. The V4L2 device does offer hardware accelerated JPEG and MJPEG streaming. Is there a reason why you're choosing not to use it?
Even more bizarre, the code appears to be doing a conversion from the YUV420/YUYV that is captured to RGB. That means the JPEG library will have to be converting back to YUV again to encode the image.

I think my recommendation here is to either do something a little smarter in your app to ask for MJPEG, or to reduce the resolution to 720P so that it goes through the video capture path. I will make a note to have a look at exposure issues on multiple still captures, but I haven't got time to look into it now - sorry.

Re: image goes black after a couple of minutes

Posted: Tue Jan 28, 2014 12:24 pm
by flok
6by9 wrote:1920x1080 YUYV will be going through the stills capture path - that'll limit the max capture rate apart from anything else.
I can't remember the full detail of how the exposure calcs run during stills capture. James is correct that most of the adjustment happens when preview is running, so multiple stills captures (JPEG, or YUV/YUYV/RGB resolution over 720P) may hit issues.
Any ways around that?
Reading the main text from http://www.vanheusden.com/koudevoeten/
Koudevoeten is a small application which streams (MJPEG stream) the video stream of any video4linux device via it's build-in webserver. MJPEG streams can be viewed with most webbrowsers (apart from internet explorer, of course).
If it's asking for YUYV then it must be doing the JPEG/MJPEG encode on the ARM. The V4L2 device does offer hardware accelerated JPEG and MJPEG streaming. Is there a reason why you're choosing not to use it?
Koudevoeten is originally a generic video4linux capturing application. It is only since yesterday that I tried getting it to work on the Raspberry Pi.
Even more bizarre, the code appears to be doing a conversion from the YUV420/YUYV that is captured to RGB. That means the JPEG library will have to be converting back to YUV again to encode the image.
Yes, that's an optimalization I'm pondering about.
The reason it was done this way is to have a generic pixel format available to "consumers" of the source-object: that way I don't need multiple implementations for e.g. VNC support.
I think my recommendation here is to either do something a little smarter in your app to ask for MJPEG, or to reduce the resolution to 720P so that it goes through the video capture path. I will make a note to have a look at exposure issues on multiple still captures, but I haven't got time to look into it now - sorry.
Ok I'll add a note to the website regarding the 720P work-around.
Please note that this problem is probably not specific for koudevoeten: other video4linux applications will hit this problem as well (unless they use the 720P resolution as you suggested).
[...]
I verified that when using 1280x720 there's no problem with exposure.
Is the exposure problem something I can override/fix with a v4l2 call?
Especially since when using 1280x720 not the whole image is available: part of it is cut-off(!).

regards

Re: image goes black after a couple of minutes

Posted: Tue Jan 28, 2014 2:02 pm
by 6by9
flok wrote:
I think my recommendation here is to either do something a little smarter in your app to ask for MJPEG, or to reduce the resolution to 720P so that it goes through the video capture path. I will make a note to have a look at exposure issues on multiple still captures, but I haven't got time to look into it now - sorry.
Ok I'll add a note to the website regarding the 720P work-around.
Please note that this problem is probably not specific for koudevoeten: other video4linux applications will hit this problem as well (unless they use the 720P resolution as you suggested).
[...]
I verified that when using 1280x720 there's no problem with exposure.
Is the exposure problem something I can override/fix with a v4l2 call?
Especially since when using 1280x720 not the whole image is available: part of it is cut-off(!).

regards
No, there's no way to fix this via V4L2 except possibly running the overlay at the same time as your captures.

The field of view difference is being looked at, but the settings provided by the sensor manufacturer just aren't working at the moment.
The issue is that the sensor can not stream the full image at 30fps, so only reads out the central 1920x1080 pixels from the middle of the 2592x1944 array. The new settings are meant to subsample both horizontally and vertically on the sensor to read out the whole frame as 1296x972. The downside is that this is then only just greater than 720P, so 1080P recording will always need to be cropped with this sensor.

Re: image goes black after a couple of minutes

Posted: Tue Jan 28, 2014 3:36 pm
by jamesh
I think its probably worth using the binned mode for 1080p as well, and just scaling it up a little at the end. Doubt there will be much difference in image quality - less than 100 pixels vertically.

Re: image goes black after a couple of minutes

Posted: Tue Jan 28, 2014 3:43 pm
by flok
jamesh wrote:I think its probably worth using the binned mode for 1080p as well, and just scaling it up a little at the end. Doubt there will be much difference in image quality - less than 100 pixels vertically.
It is not neccessarily quality I miss, but parts of the room I try to monitor which is unfortunate.

Re: image goes black after a couple of minutes

Posted: Tue Jan 28, 2014 3:59 pm
by jamesh
Yes, it's a known problem, and we are looking at it. Omnivision have got back, so hopefully some movement on this soon.

Re: image goes black after a couple of minutes

Posted: Tue Jan 28, 2014 4:10 pm
by flok
thank you

Re: image goes black after a couple of minutes

Posted: Tue Jan 28, 2014 5:51 pm
by towolf
jamesh wrote:I think its probably worth using the binned mode for 1080p as well, and just scaling it up a little at the end. Doubt there will be much difference in image quality - less than 100 pixels vertically.
If you do that please make it explicit what is happening behind the scenes.

If you can generate roundabout 720p from binned output then what we get is 720p. Scaling that up and calling it 1080p is kinda akin to deceptive packaging.

BTW, in Germany public TV in HD runs at 720p50, they opted for this instead of 1080i50 when they had to decide due to their bandwidth alotment, and I agree with that decision.

Personally, I'd prefer no scaling at all (besides the 2x2 binning).

Re: image goes black after a couple of minutes

Posted: Tue Jan 28, 2014 10:11 pm
by jamesh
I think you will be surprised to know how many phone manufacturers use binned then scaled modes with no real loss of definition, and say it's 1080p. It's certainly slighter higher definition than 720p at 1296x972 compared to 1280x720.

But it is a tradeoff - do you want full FOV 1080p or not?

UK Freeview HD broadcast are also 720p, not sure of the frame rate.

Re: image goes black after a couple of minutes

Posted: Wed Jan 29, 2014 3:58 am
by towolf
jamesh wrote:I think you will be surprised to know how many phone manufacturers use binned then scaled modes with no real loss of definition, and say it's 1080p. It's certainly slighter higher definition than 720p at 1296x972 compared to 1280x720.

But it is a tradeoff - do you want full FOV 1080p or not?
Personally, I’m not insisting on 1080 for anything. I would love full FoV at 1296x972 px on output. And then I could, if needed, use a ROI or crop to 1280x720, I suppose.

Re: image goes black after a couple of minutes

Posted: Wed Jan 29, 2014 4:16 am
by gordon77
jamesh wrote:I think you will be surprised to know how many phone manufacturers use binned then scaled modes with no real loss of definition, and say it's 1080p. It's certainly slighter higher definition than 720p at 1296x972 compared to 1280x720.

But it is a tradeoff - do you want full FOV 1080p or not?

UK Freeview HD broadcast are also 720p, not sure of the frame rate.
Not all is 720p
http://www.bbc.co.uk/faqs/tv/bbchd_channels_schedule

And maybe most are 1080
http://www.avforums.com/threads/all-fre ... g.1530548/

Re: image goes black after a couple of minutes

Posted: Wed Jan 29, 2014 9:27 am
by jamesh
gordon77 wrote:
jamesh wrote:I think you will be surprised to know how many phone manufacturers use binned then scaled modes with no real loss of definition, and say it's 1080p. It's certainly slighter higher definition than 720p at 1296x972 compared to 1280x720.

But it is a tradeoff - do you want full FOV 1080p or not?

UK Freeview HD broadcast are also 720p, not sure of the frame rate.
Not all is 720p
http://www.bbc.co.uk/faqs/tv/bbchd_channels_schedule

And maybe most are 1080
http://www.avforums.com/threads/all-fre ... g.1530548/
Ah, interesting, I was under the impression it was all 720p.

Re: image goes black after a couple of minutes

Posted: Wed Jan 29, 2014 6:51 pm
by gordon77
Maybe you need to check any settings on your TV :D

Re: image goes black after a couple of minutes

Posted: Wed Jan 29, 2014 8:42 pm
by jamesh
That runs in 1080p always, the Freeview HD PVR (Humax) always scales up to 1080 what ever the input. So you really don't know what the source is unless you check.

Re: image goes black after a couple of minutes

Posted: Thu Jan 30, 2014 8:20 pm
by jbeale
jamesh wrote:But it is a tradeoff - do you want full FOV 1080p or not?
I think if people could see still frame captures that show the difference (or lack of it) in this case, they might be less prone to argue just on counting pixels. I suspect there isn't much useful information to be gained by "true" 1080p vs scaled from this sensor, but that's just a suspicion. I also suspect in low-light you could get significantly more true image per-pixel information using binned+scaled, over cropped.

Re: image goes black after a couple of minutes

Posted: Sun Feb 02, 2014 5:02 pm
by flok
Is capturing jpeg frames via video4linux known to be working?
Because I get only black frames in valid(!) jpeg format.
This is both in the 1280x720 as well as the 1920x1080 resolution.
The frames are immediately black, not after a while.
This is a console/terminal application without any preview windows.

Re: image goes black after a couple of minutes

Posted: Sun Feb 02, 2014 5:21 pm
by jamesh
I think probably its not had enough time to expose the image correctly again. Try it with a preview and give it a second to gets its settings right.

Re: image goes black after a couple of minutes

Posted: Sun Feb 02, 2014 5:30 pm
by flok
Hi James,

The device is constantly open, e.g. I constantly fetch frames from it so it should give an image if it was the issue you described.

Also: the device is used in a remote location (birdhouse but friends of mine would like to use it as a security cam) so I cannot run an X environment (for the preview) on it.