Raspiraw - High FPS ~320x240 with full FoV - possible?

[TWhidden]

Hello,

Following @6by9 and @HermannSW posts using raspiraw/+fork -- Amazing work, just stuck on getting the FoV.

Can I use raspiraw, to capture the entire sensor image area (Field of View), but skip H and V sensor lines, to get it closer to a 320x240ish Bayer image? It seems when I mess with the modes or manual resolutions, I get the smaller resolution, but I also reduce the FoV.. I want the same FoV, but just lesser resolution.

I am working a very close up image (~13mm from the v2 camera). I need the whole sensor image, but not all the resolution, and at a high frame rate (very fast moving elements that cant be blurred). I want to capture less data from the sensor, but need the full view of the sensor. I am using IR Diodes to illuminate if that helps/hurts - this is in a zero light area (looks great in raspivid)

Can someone recommend to me a command line configuration to test/try to validate ? I tried the --vinc and --hinc, but haven't had much luck (or I'm doing it wrong).

Thanks!

Travis

[HermannSW]

Which camera version do you have?

With 8MP v2 you are lost, there is only two horizontal and two vertical offsets. So you can get FOV(640x480) for 320x240, but not more.

5MP v1 camera (available as 5$ China clone these days only) has full FOV for 640x480 and should allow for 320x240 with full FOV (I don't have Pi with me to try).

Here are v1 camera FOVs:
https://picamera.readthedocs.io/en/rele ... 2/fov.html


and v2 FOVs:


In case you have a v2 camera not everything is lost, you can remove original v2 lens, add M12 mount and then use a wide angle M12 lens to increase FOV, more details here:
viewtopic.php?f=63&t=228284&p=1400452#p1400452

P.S:
While v1 camera can record 640x480@90fps "only", it can capture 320x240@180fps, with raspiraw as well as with raspivid (you have to add 320x240 mode line and compile your version with 320x240 as mode 8. I did that in the past, but am not sure whether v1 camera 320x240 FOV was full or not).

P.P.S:
With v2 camera you can record 320x240@360fps, but only with raspiraw. Because of a GPU bug that will not be fixed maximal framerate for raspivid is 200fps (and only for 640x480 mode, for other modes it is 120fps). With I2C command injection it is possible to run self compiled raspivid 320x240@200fps.

[TWhidden]

Very informative. Thanks so much for supporting the community like you do. I have spent quite a bit of time reading all these threads that become part of history with the RP projects.

For the 640x480 FoV -- can you give me a command line example to experiment with? I am currently using your fork.

As for hardware: Currently am using the v2 NoIR camera, but I can buy another one if you recommend such as the V1 camera. I also bought a Omnivision 5647 (https://www.amazon.com/gp/product/B00N1YJKFS/) but have not hooked it up to try, mainly because I can get what I need with the V2 in the full view.

The M12 I was considering doing also, but I have a very limited vertical space in the application I am putting together, so trying to use the v2 if possible.

Main goal, capture a fast moving item passing past the camera with as little blur as possible so I can use tensorflow ML to classify/identify. The low resolution is fine as they are just symbols and numbers that are quite large when passing by the camera when this close.

[HermannSW]

I still have no access to Pi and Pi camera, will be home tomorrow evening.
I just found 320x340@180fps tool for v1 camera:
https://github.com/Hermann-SW/fork-rasp ... ls/320x240

It has vinc 1F, while 640x480 tool has vinc 17 -- therefore that v1 320x240 tool has full FOV.
I just did recursive search in my Raspberry folder to find a 320x240 v2 tool -- there is none yet.
Perhaps I did not build one because v2 camera allows to get 640x480@180fps with raspivid and that looks much better.


I started raspivid high framerate work (in the beginning 90fps was max fps for v1, 120fps was max fps for v2 camera even with raspiraw, now it is 750fps/1007fps) because I intercepted raspiraw for frame analysis and robot control. What I have achieved already with that is automatic camera tilt calibration:
viewtopic.php?f=43&t=189661#p1231151



Recently I did reverse engineer Wifi protocol between Eachine E52 drone and android control app:
https://github.com/Hermann-SW/wireless- ... -E52-drone

I was able to tell the drone to send its camera h264 video stream to Raspberry and process there further.
At least for letting the the onboard Pi Zero W flying the drone autonomously I will have to learn how to process single frames from h264 stream:



Back to your v2 camera FOV problem -- couldn't you cover the same FOV with reduced 640x480FOV by moving v2 camera a bit further away from the area you want to record?


P.S:
> Main goal, capture a fast moving item passing past the camera with as little blur as possible
>
In addition to high framerate you will want to set shutter time to less than what would be possible for a given framerate to avoid the blur.
I once went down to 200us(!) shutter time for capturing 20140rpm RC airplane propeller for minimizing rolling shutter effect:
https://lb.raspberrypi.org/forums/viewt ... 4#p1319597


This is from 640x75@1007fps v2 camera video, animation plays at 1fps:


This is 640x150_s tool 1007fps recording of same scene

[HermannSW]

I thought again -- while I cannot make 320x240 FOV bigger than the 640x480 FOV shown in previous posting, I can double the 640x480 FOV for you!
I did just capture the 640x480 FOV area from previous v2 camera photo, doubled it in both domensions and inserted again. As you can see with that 640x480 mode you can get nearly full camera FOV!


And you can take 640x480@180fps with v2 camera (while v2 camera can capture 320x240@360fps).
In case 180fps is good for you, taking 640x480 with doubled FOV should be the way to go.

[TWhidden]

As for distance -- can't by much due to physical limitations. Maybe 3-5 more MM I could do. I have considered turning the camera sideways and using a mirror for this -- but that also has some challenges as the V2 camera on its side is to high for for the device container its going in.

Can you tell me the switches you used for the 640x480 almost FOV shown above? Or checkin another tool in your repo for reference. I want to get started with that tonight. That might work out OK.

Thanks!

[HermannSW]

Can you tell me the switches you used for the 640x480 almost FOV shown above?

No switches, as I said I did image manipulation only with Gimp.

I will look into 640x480 double FOV as well as 320x240 tools in the evening when at home.

I found the v2 tools (all 640xH) for v2 camera, they are in v2 directory of .zip drop from this posting:
viewtopic.php?t=213482#p1320044

[6by9]

There is a 4x4 binning mode. https://github.com/6by9/raspiraw/pull/21 is a nasty hack that has been proposed to enable it, but I haven't had a chance to test it out. That should double the FOV of the mode.
The main downside of that is that the datasheet says it is a digital binning mode, so it won't provide the normal improvement in noise behaviour that analogue binning provides.

[HermannSW]

On the train I played a bit with LibreOffice Calc.
I assumed that doubling FOV by "4x4 binning" or "hvinc3" (horizontal/vertical increment: 1/1->3/3) are independent.
So I created table starting with target resolution on left side (320x240/640x480) and computing the effective FOV with 2x2 binning, either 4x4 or hvinc3 and 4x4 plus hvinc3:


Cyan background is full FOV of 8MP v2 camera, yellow background is "nearly full".

For middle group I came the other way around, dividing 3280x2464 by 4 and 8.
Since it is not clear without experiment that 820x616 really can be done at 140fps, reduced 640x616 format was added as well.

Finally I moved to nearest standard format 800x600 and 400x300 resulting in yellow background.

Plan is that I will add 4x4 binning option to fork-raspiraw, and then provide 7 new tools for the 7 target resolutions shown above and test whether the projected framerates in most right column can actually be achieved (and whether 4x4 binning and hvinc3 are independent).

[HermannSW]

I got side tracked by creating a millisecond resolution laser clock and taking 1007fps 640x150_s video of that in another thread:
viewtopic.php?f=43&t=228145&p=1401287#p1401745



Now I took 640x480 tool from v2 folder of code drop I pointed to and increased verrtical and horizontal increments from 1/1 to 3/3.
I did print the original v2 camera FoV image and captured with 640x480 tool:



Then I used new tool 640x480_B:



This is the diff between both tools (both tools are atached for you):

Code: Select all

< raspiraw -md 7 -t $1 -ts tstamps.csv -hd0 hd0.32k --height 480 --top 0 --fps $fps -sr 1 -o /dev/shm/out.%04d.raw 2>/dev/null >/dev/null
---
> raspiraw -md 7 -t $1 -ts tstamps.csv -hd0 hd0.32k --height 480 --voinc 03 --hoinc 03 --fps $fps -sr 1 -o /dev/shm/out.%04d.raw 2>/dev/null >/dev/null

Working "--left" option is needed for v2 camera. And there should be more than just white on right half. So it is not so easy as I thought to get 640x480 recording with FoV(1280x960). Also the colors in top left quarter seem to be wrong.

[TWhidden]

I also see the same issue. The image is in the top left quarter of the image.

[HermannSW]

Adding "--width 320 --height 240" to 640x480_B tool gives 640x480_s tool.
Because of voinc/hoinc the captured 320x240 image covers same FOV as 640x480 does:


Removing voinc/hoinc gives 320x240 tool, that just captures 320x240 frame starting top left in 640x480 frame:


Need to figure out why voinc/hoinc "works" after halving resolution, but does not without.

[HermannSW]

This is a big step into full FoV high framerate v2 camera direction.

I started with 1280x720 tool, inserted "--width=640 --height=360 --voinc=03 --hoinc=03" and got this 640x360 image with double FOV in both directions (78%/58% of full FoV):


Only drawback is that it is not 640x480 (yet), but what you get with 1280x720_s tool is 640x360@240fps !
This makes sense because v2 camera can do 640x480@180fps, and 180*(480/360)=240.

Find image taken with 1280x720 tool for comparison, as well as 1280x720_s tool in this archive (too big to attach in forum):
https://stamm-wilbrandt.de/en/forum/1280x720_s.zip


P.S:
I started from 1640x1232, 1640x922 and 1920x1080 tools as well, but "--voinc 03" and "--hoinc 03" have no effect for those modes.

[HermannSW]

4x4 binning is interesting, I added "--bin44" option to raspiraw on my fork-raspiraw with this commit:
https://github.com/Hermann-SW/fork-rasp ... 1f194b9ead

I took all but the "bonus" of pull request 6by9 pointed to before.

Initial experiment was to just capture 640x480 with 4x4 binning, and it works.
This is 640x480 frame from video captured with newly checked in tool 640x480_44:


There are good and bad news, good news first:

• FoV doubled in both directions, now 78% of full FoV in both directions

Bad news:

• for some reason no more than 90fps is possible although "--hoinc 03 --voinc 03" can do double that framerate when considering vertical size
• same maximal framerate as "1280x720" tool
• for now combination of "--bin44" and "--hoinc 03 --voinc 03" seems not to work, most likely because 640*4*2 > 3280