Raw sensor access / CSI-2 receiver peripheral

[rayzrocket]

I can't wait to get some C code running using the CSI-2 raspicamV2. For now my skills limit me to basic Python.
My Python3 code with CV2 frame capture on a USB camera and Numpy array manipulation running in Thonny IDE....
Track algo of Minimum sum:
2ms / 25 pixel wide area
9ms / 90 pixel wide area
12ms / 100 pixel wide area
15ms / 115 pixel wide area
20ms / 140 pixel wide area

Placing text on the frame = 2 to 3 ms
Read 25 bytes on Pi serial input @ 100kbaud = 3ms
Parse SBUS protocol into useful values = <1ms
Send out 25 bytes = 3ms
cv2.imshow (show the image on the screen) = <5ms
I sent you email of my algo and code since it's a bit large for the forum.

[HermannSW]

Found this tweet from @MachinePix:

Removing unripe tomatoes with an optical sorter.

https://twitter.com/MachinePix/status/9 ... 5057209344

Not sure what optical sensors are used, nor have any information on the framerate in case it is a/are camera(s).
This is just small clip from slow motion part of the tweet's video, raspiraw should be able to provide support for the same functionality:

[lak4cyut]

The main ones:
- Licence. Any kernel driver almost has to be GPLv2 and therefore open source. Check with your sensor supplier that they are happy for the register set to be released. Copying raspiraw means that the register set can be hidden in your userspace app which can be under any licence you fancy (including closed). It's all a little silly as it is trivial to put an I2C analyser on the lines to the camera module, run your app, and capture the relevant commands.
- Framework. V4L2 is the Linux standard APIs and is therefore portable to other platforms. MMAL is specific to the Pi.
- Support. I'm less inclined to fix any issues discovered in rawcam than in V4L2. V4L2 is seen as the way forward.

In the first method, can you give me an example of drive file? Could I just modify the ov5647.c(/drivers/media/i2c/ov5647.c) to other sensor?

/drivers/media/i2c/ov5647.c is a relatively basic driver, but would work. imx258.c is a little more comprehensive, but lacks the dt/fwnode configuration and regulator control (not essential if you set the power control GPIO some other way). I believe ov5640.c is a reasonable example of that.

How to use the v4l2 driver to get raw data from sensor? Is it possible to transfer raw10/raw12 to yuv data?(use arm core?)

You use the V4L2 API to get raw frames out.
"v4l2-ctl --stream--map=3 --stream-to=foo.raw --stream-count=100" would be a simple existing tool.
https://linuxtv.org/downloads/v4l-dvb-a ... ure.c.html is the standard example for grabbing frames. Do what you want in process_image(). You want to be using the MMAP method, not read (userptr isn't supported).

Hi 6by9,
Thanks for your information.
I built a custom kernel to enable bcm2835-unicam and ov5647 kernel modules and add ov5647 dt overlay, try to capture camera image via V4L2 official interface. (Of course I bought a Raspberry Pi official camera module - OV5647 version.)
And use v4l2-ctl to capture image.

Code: Select all

v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=frame.raw --stream-count=1

But the captured image is abnormal (overlay & distortion) (I put the image below).
Would you like to provide me some hint about how to debug this problem? Or do you know what caused it.
It drive me crazy...

Thanks for the help.

frame2.jpg (79.79 KiB) Viewed 2005 times

frame.jpg (122.32 KiB) Viewed 2005 times

Detail info:
Kernel base version: 4.19.64
Raspberry Pi: 3 model B v1.2
Camera Module: Raspbery Pi (OV5647) Rev 1.3

[6by9]

Hi 6by9,
Thanks for your information.
I built a custom kernel to enable bcm2835-unicam and ov5647 kernel modules and add ov5647 dt overlay, try to capture camera image via V4L2 official interface. (Of course I bought a Raspberry Pi official camera module - OV5647 version.)
And use v4l2-ctl to capture image.

Code: Select all

v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=frame.raw --stream-count=1

But the captured image is abnormal (overlay & distortion) (I put the image below).
Would you like to provide me some hint about how to debug this problem? Or do you know what caused it.
It drive me crazy...

Thanks for the help.

Duplicate of https://www.raspberrypi.org/forums/view ... 2#p1522892, therefore I'm not responding here (it's not relevant to rawcam).

[HermannSW]

@6by9 I want to play with my new 4GB Pi4B and raspiraw.
v2 camera works with raspivid on Pi4B.
I did clone latest raspiraw from your repo, because that knows about Pi4B.
But camera shows up as /dev/i2c-1 and not /dev/i2c-0.
I did change camera_i2c last line in 4B section to "i2cdetect -y 1" because of that.
But it shows no connected device, just double dashes everywhere.
I added "-y 1" to raspiraw call in tools/640x128, but the command fails as well.

Is this a Buster does not like raspiraw thing?
What do I need to change/do differently in order to get raspiraw working on Pi4B?

[HermannSW]

Sorry, my fault, I only enabled I2C in raspi-config and missed to add "dtparam=i2c_vc=on" to "/boot/config.txt".
Now raspiraw works, but I get two super huge frame skips when running 640x128 tool, investigating:

Code: Select all

pi@raspberrypi4B:~/raspiraw/c $ 640x128 2000
removing /dev/shm/out.*.raw
capturing frames for 2000ms with 349fps requested
495 frames were captured at 351fps
frame delta time[us] distribution
      1 -9223372036854775808
      1 
      3 0
      1 2839
      1 2841
      3 2842
    339 2843
    124 2844
      1 2848
      1 5686
      1 5687
      1 14218
      1 19903
      1 22746
      1 22747
      2 25589
      2 25590
      3 28432
      1 28433
      1 36963
      2 51179
      2 54022
      1 105201
      1 9223372036723459902
after skip frame indices (middle column)
5686,2,129328465
19903,3,129348368
54022,7,129482001
51179,8,129533180
105201,9,129638381
51179,11,129717992
54022,12,129772014
36963,16,129888588
14218,17,129902806
5687,139,130317921
9223372036723459902,491,-9223372036854775808
2% frame skips
pi@raspberrypi4B:~/raspiraw/c $

"head" looks fine:

Code: Select all

pi@raspberrypi4B:~/raspiraw/c $ head -3 tstamps.csv 
,1,129322779
5686,2,129328465
19903,3,129348368
pi@raspberrypi4B:~/raspiraw/c $

But "tail" looks ugly:

Code: Select all

pi@raspberrypi4B:~/raspiraw/c $ tail -6 tstamps.csv 
2844,490,131315906
9223372036723459902,491,-9223372036854775808
-9223372036854775808,492,0
0,493,0
0,494,0
0,495,0
pi@raspberrypi4B:~/raspiraw/c $ 

This is all to get raspiraw running:

Code: Select all

pi@raspberrypi4B:~ $ cat /etc/modules-load.d/modules.conf 
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

i2c-dev
pi@raspberrypi4B:~ $ 

Code: Select all

pi@raspberrypi4B:~/raspiraw/c $ grep i2c /boot/config.txt 
dtparam=i2c_arm=on
dtparam=i2c_vc=on
pi@raspberrypi4B:~/raspiraw/c $ 

[HermannSW]

Pi4B raspiraw from this branch showed problems:
https://github.com/6by9/raspiraw

So I used this repo for testing how long high framerate video becomes possible with Pi4B 2GB /dev/shm filesystem:
https://github.com/Hermann-SW/fork-raspiraw

The frame drops are higher on Pi4B than on the other PIs, I have no explanation.
I reduced the request framerate so long that finally 0% frame skips (integer division, means less than 1%) are reported.

For 640x75 tool framerate had to be reduced to 856fps (from 1007fps):

Code: Select all

...
2337,30156,12983691033
2337,30500,12984094153
0% frame skips
pi@raspberrypi4B:~/fork-raspiraw/c $ df /dev/shm
Filesystem     1K-blocks    Used Available Use% Mounted on
tmpfs            1973500 1953472     20028  99% /dev/shm
pi@raspberrypi4B:~/fork-raspiraw/c $ 

36 seconds of 640x75 video fit into /dev/shm:

Code: Select all

pi@raspberrypi4B:~/fork-raspiraw/c $ ./640x75 36000
removing /dev/shm/out.*.raw
capturing frames for 36000ms with 850fps requested
30523 frames were captured at 856fps
frame delta time[us] distribution
      1 
      1 1162
      1 1163
      7 1164
     23 1165
    550 1166
   1117 1167
  14349 1168
  12640 1169
   1116 1170
    509 1171
     14 1172
      7 1173
      1 1174
      3 2334
      6 2335
     62 2336
     69 2337
     11 2338
      1 4672
      4 4673b
      8 4674
     17 5842
      3 5843
      1 7010
      2 7011
after skip frame indices (middle column)
7011,2,12948131142
2335,384,12948578664
5842,450,12948660456
2337,552,12948780809
2336,625,12948867275
2336,895,12949183929
7010,1313,12949678189
2337,1666,12950091826
2336,1910,12950378101
2337,1923,12950394459
2336,1995,12950479756
4673,2175,12950693585
2337,2177,12950697091
2336,2680,12951285997
2336,2693,12951302355
5843,3038,12951710150
2336,3364,12952092238
2337,3464,12952210253
4674,3901,12952724377
2337,3978,12952815518
2336,4048,12952898478
2337,4235,12953118150
2337,4579,12953521270
2337,4650,12953605400
2336,4734,12953704719
2337,4749,12953723415
5842,4761,12953742110
2337,5089,12954126535
2336,5417,12954510960
2337,5519,12954631312
5842,5623,12954757506
2337,5859,12955034432
2337,6013,12955215545
2337,6014,12955217882
2337,6029,12955236578
4673,6484,12955771733
2336,6629,12955942329
2336,6696,12956021786
2336,6782,12956123441
2337,7144,12956547594
5842,7345,12956787129
2336,7466,12956929682
2337,7915,12957455491
2337,8087,12957657637
2336,8153,12957735923
4672,8208,12957803694
2336,8427,12958060756
2336,8685,12958363388
2336,8837,12958542164
2337,8942,12958666021
4674,9070,12958819090
2336,9197,12958968653
2337,9455,12959271287
2335,9520,12959348404
2337,9712,12959573918
5842,9930,12959833317
2336,9966,12959876550
2338,10116,12960052990
2336,10117,12960055326
2336,10200,12960153477
2337,10221,12960179183
2337,10479,12960481815
5842,10789,12960848713
2336,10992,12961087080
2336,11250,12961389712
2337,11508,12961692346
2336,11570,12961765958
4674,11653,12961866446
2337,11762,12961994977
2337,12020,12962297610
2336,12167,12962470543
2336,12253,12962572199
2337,12276,12962600242
5842,12513,12962881842
2336,12530,12962902874
2336,12615,12963003362
2338,12848,12963276784
2336,12934,12963378440
2336,13044,12963508139
5842,13373,12963897238
2337,13533,12964085362
2337,13556,12964113405
2337,13557,12964115742
2336,13615,12964184680
2336,13812,12964416036
4674,14235,12964913803
2337,14236,12964916140
2336,14299,12964990921
2337,14324,12965021301
2336,14582,12965323933
2336,14986,12965797162
5842,15096,12965930367
2336,15182,12966032023
2338,15583,12966501747
2336,15668,12966602234
2337,15697,12966637288
5842,15955,12966943426
2337,16127,12967145570
2336,16299,12967347715
2336,16350,12967408475
2337,16642,12967750835
5842,16814,12967956485
2335,17034,12968214715
2337,17154,12968356100
2334,17412,12968658732
2338,17634,12968919301
2337,17635,12968921638
2337,17668,12968961366
5843,17675,12968974219
2336,17714,12969020956
2337,17921,12969263997
2336,18179,12969566629
2338,18314,12969725542
2336,18350,12969768775
2336,18399,12969827197
2337,18434,12969869262
5843,18533,12969989614
2336,18688,12970171894
2336,19082,12970633438
2337,19204,12970777159
4673,19396,12971005009
2337,19459,12971079791
2337,19717,12971382425
2336,19765,12971439679
2337,19974,12971685056
4674,20258,12972020406
2336,20450,12972245919
2337,20487,12972290321
2336,20745,12972592953
5842,21122,12973038139
2336,21132,12973050992
2334,21257,12973198218
2337,21774,12973803484
2337,21775,12973805821
2336,21818,12973857232
4674,21982,12974052366
2335,22027,12974106115
2336,22503,12974663473
2337,22543,12974711380
2334,22801,12975014012
5842,22844,12975068930
2337,22882,12975114501
2338,23098,12975368058
2336,23184,12975469714
2337,23311,12975619277
2337,23655,12976022398
5842,23704,12976084326
2338,23780,12976174299
2337,23822,12976224543
2335,23865,12976275954
2337,24424,12976930295
2336,24553,12977082195
5842,24566,12977102059
2337,24591,12977132439
2337,24935,12977535560
2336,25236,12977888436
2336,25365,12978040336
4674,25427,12978116287
2337,25706,12978443457
2337,25878,12978645602
2336,25919,12978694677
5842,26290,12979132851
2337,26476,12979351354
2337,26603,12979500918
2337,26992,12979956619
5842,27152,12980148247
2337,27287,12980307159
2337,27418,12980461396
2337,27762,12980864516
2338,27934,12981066662
2336,27973,12981113399
4674,28013,12981163643
2335,28533,12981772413
2336,28658,12981919640
2337,28704,12981974558
7011,28875,12982180208
2337,29217,12982580991
2338,29253,12982624225
2338,29254,12982626563
2336,29338,12982725881
2336,29386,12982783135
2337,29730,12983186256
4673,29733,12983193266
2338,29935,12983430466
2336,30021,12983532121
2337,30156,12983691033
2337,30500,12984094153
0% frame skips
pi@raspberrypi4B:~/fork-raspiraw/c $ df /dev/shm
Filesystem     1K-blocks    Used Available Use% Mounted on
tmpfs            1973500 1953472     20028  99% /dev/shm
pi@raspberrypi4B:~/fork-raspiraw/c $ 

This is the tool:

Code: Select all

pi@raspberrypi4B:~/fork-raspiraw/c $ cat 640x75
#!/bin/bash

if [ "$1" = "" ]; then echo "format: `basename $0` ms"; exit; fi

echo "removing /dev/shm/out.*.raw"
rm -f /dev/shm/out.*.raw

fps=850
echo "capturing frames for ${1}ms with ${fps}fps requested"
raspiraw -md 7 -t $1 -ts tstamps.csv -hd0 hd0.32k --height 75 --voinc 01 --fps $fps -sr 1 -o /dev/shm/out.%04d.raw 2>/dev/null >/dev/null

us=`cut -f1 -d, tstamps.csv | sort -n | uniq -c | sort -n | tail -1 | cut -b9-`
l=`ls -l /dev/shm/out.*.raw | wc --lines`
echo "$l frames were captured at $((1000000 / $us))fps" 

echo "frame delta time[us] distribution"
cut -f1 -d, tstamps.csv | sort -n | uniq -c

echo "after skip frame indices (middle column)"
grep "^[2-9]" tstamps.csv

skips=`grep "^[2-9]" tstamps.csv | wc --lines | cut -f1 -d\ `
stamps=`wc --lines tstamps.csv | cut -f1 -d\ `
per=`expr \( 100 \* $skips \) / \( $skips + $stamps \)`
echo "$per% frame skips"
pi@raspberrypi4B:~/fork-raspiraw/c $ 

Reduction of framerate to 615fps (from 667fps) was necessary to get 0% frame skip reporting:

Code: Select all

...
3233,19469,12567260770
3231,19479,12567278549
0% frame skips
pi@raspberrypi4B:~/fork-raspiraw/c $ df /dev/shm
Filesystem     1K-blocks    Used Available Use% Mounted on
tmpfs            1973500 1961200     12300 100% /dev/shm
pi@raspberrypi4B:~/fork-raspiraw/c $ 

In total 32 seconds of 640x128 video can be captured into /dev/shm:

Code: Select all

raspberrypi4B:~/fork-raspiraw/c $ ./640x128 32000
removing /dev/shm/out.*.raw
capturing frames for 32000ms with 615fps requested
19612 frames were captured at 618fps
frame delta time[us] distribution
      1 
      6 1611
     13 1612
     10 1613
     19 1614
     78 1615
  11904 1616
   7372 1617
     30 1618
     16 1619
      8 1620
     15 1621
      3 1622
      1 3229
      6 3231
     49 3232
     50 3233
      1 4848
     14 4849
      9 6465
      7 6466
after skip frame indices (middle column)
6465,494,12536299075
3233,637,12536531834
3233,669,12536585175
3233,678,12536601339
3232,692,12536625584
3233,694,12536630433
3232,695,12536633665
4849,1112,12537310927
3233,1128,12537338406
3232,1181,12537425690
3233,1182,12537428923
3233,1183,12537432156
3232,1186,12537438621
3232,1684,12538245193
6465,1731,12538326011
3232,2169,12539035601
3233,2174,12539045300
3233,2177,12539051766
6465,2353,12539341096
3232,2529,12539627194
3232,2653,12539829242
3233,2654,12539832475
3233,2655,12539835708
3231,2668,12539858336
4849,2973,12540354564
3233,3115,12540585707
3232,3145,12540635814
3233,3456,12541140123
4848,3596,12541369648
3233,3636,12541435921
3233,3637,12541439154
3233,3638,12541442387
3232,3656,12541473097
3233,4096,12542185920
3232,4126,12542236027
3232,4128,12542240876
3232,4132,12542248958
3232,4150,12542279669
4849,4214,12542386349
3233,4565,12542955314
3233,4626,12543055531
3233,4643,12543084625
4849,4836,12543399817
3232,5109,12543842705
3233,5110,12543845938
3232,5137,12543891196
4849,5459,12544414902
3233,5570,12544595937
3232,5599,12544644428
3233,5600,12544647661
3233,5629,12544696153
6465,6080,12545429987
3231,6124,12545502723
3233,6587,12546252723
3233,6588,12546255956
3232,6604,12546283434
3231,6619,12546309295
6465,6701,12546446687
3233,7071,12547046364
3233,7072,12547049597
3233,7074,12547054446
3232,7095,12547090006
3232,7110,12547115868
3233,7218,12547292053
4849,7320,12547460155
3232,7589,12547896578
3232,7604,12547922440
4849,7944,12548475240
3232,8055,12548656275
3233,8056,12548659508
3233,8082,12548703150
3233,8083,12548706383
3232,8096,12548729012
3233,8548,12549461231
6465,8563,12549490325
3232,8590,12549535583
6466,9186,12550503793
3233,9529,12551059827
3232,9531,12551064676
3233,9582,12551148728
6466,9807,12551517261
3233,9937,12551729006
3233,10018,12551861550
3233,10020,12551866399
3232,10059,12551931054
3233,10073,12551955300
4849,10427,12552530729
3232,10554,12552737626
3232,10568,12552761872
6466,11051,12553547431
3232,11063,12553568443
3232,11547,12554352386
6466,11675,12554564131
3232,12042,12555158958
3229,12055,12555181587
4849,12298,12555577599
3233,12525,12555946134
3233,12550,12555988160
4849,12921,12556591067
3233,13016,12556746240
3232,13021,12556755938
3233,13044,12556794732
3232,13508,12557546346
3231,13541,12557601302
4849,13542,12557606151
3232,14024,12558386862
6465,14166,12558621236
3232,14519,12559193434
3232,14531,12559214447
4849,14789,12559634704
3233,15014,12560000006
3233,15026,12560021019
4849,15412,12560648172
3232,15509,12560806578
3232,15521,12560827590
3233,16006,12561613150
3232,16018,12561634162
6465,16034,12561664873
6466,16659,12562679958
3232,16971,12563185884
3233,17007,12563245691
6466,17281,12563693426
3231,17502,12564052261
4849,17905,12564706894
3232,17987,12564841054
3232,17997,12564858834
3232,18452,12565595902
3232,18470,12565626613
3232,18482,12565647626
3232,18492,12565665406
6465,18526,12565725211
3232,18971,12566446116
3233,18975,12566454198
3232,18985,12566471978
6466,19148,12566740296
3233,19469,12567260770
3231,19479,12567278549
0% frame skips
pi@raspberrypi4B:~/fork-raspiraw/c $ df /dev/shm
Filesystem     1K-blocks    Used Available Use% Mounted on
tmpfs            1973500 1961200     12300 100% /dev/shm
pi@raspberrypi4B:~/fork-raspiraw/c $ 

This is the tool:

Code: Select all

pi@raspberrypi4B:~/fork-raspiraw/c $ cat 640x128 
#!/bin/bash

if [ "$1" = "" ]; then echo "format: `basename $0` ms"; exit; fi

echo "removing /dev/shm/out.*.raw"
rm -f /dev/shm/out.*.raw

fps=615
echo "capturing frames for ${1}ms with ${fps}fps requested"
raspiraw -md 7 -t $1 -ts tstamps.csv -hd0 hd0.32k --height 128 --top 0 --vinc 17 --fps $fps -sr 1 -o /dev/shm/out.%04d.raw 2>/dev/null >/dev/null

us=`cut -f1 -d, tstamps.csv | sort -n | uniq -c | sort -n | tail -1 | cut -b9-`
l=`ls -l /dev/shm/out.*.raw | wc --lines`
echo "$l frames were captured at $((1000000 / $us))fps" 

echo "frame delta time[us] distribution"
cut -f1 -d, tstamps.csv | sort -n | uniq -c

echo "after skip frame indices (middle column)"
grep "^[2-9]" tstamps.csv

skips=`grep "^[2-9]" tstamps.csv | wc --lines | cut -f1 -d\ `
stamps=`wc --lines tstamps.csv | cut -f1 -d\ `
per=`expr \( 100 \* $skips \) / \( $skips + $stamps \)`
echo "$per% frame skips"
pi@raspberrypi4B:~/fork-raspiraw/c $ 

[HermannSW]

I wanted to watch the updating phase of a max7219 8-digit tube 7-segment display.
This was the first time I was able to use 640x64 tool to record 665fps video with v1 camera not for speed demonstration purpose.

The animations were extracted from two videos, from a phase without frame skips (2% frame skips in total for the videos).
The "8." on left is fixed, gets cleared first when printing a new value, and is directly written after printing.
So "8." displayed fully shows a real value on the right, otherwise value on right is in transition.

It takes less than 3/665=4.5ms to print all 8 digits (clear space left of number):


For 1/100th second clock example you can see updates every 6th or 7th frame:


https://github.com/Hermann-SW/DigitLedDisplay

What was remarkable is this:
For taking 665fps raspiraw video in the past I had to provide bright lighting to the scene.
Here scene around display was dark, with NO additional lighting.
The display tubes were bright enough to display nicely with only 1.5ms shutter time at 665fps.
And that was not even full brightness possible, was only 10 of 1-15 range.