6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6879
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Wed Jul 25, 2018 10:34 am

@luiscgalo: branching the conversation off to a new thread, as we've now got three interleaved conversations going on. I'll respond there.
viewtopic.php?f=43&t=218928
(I'll be doing the same thing with grimepoch's thread in a moment).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6879
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Wed Jul 25, 2018 10:56 am

@grimepoch: As with luiscgalo, branching the conversation off to a new thread.
viewtopic.php?f=43&t=218933
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6879
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Wed Jul 25, 2018 11:57 am

General comment, if you're looking at interfacing a new chip/sensor/FPGA/thingamebob in, please start a new thread and post a link on this thread.
This thread is already 20 pages long, and is likely to grow in a fairly uncontrolled manner if people try adding all sorts of random support requests on it.
Thank you.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Edeard95
Posts: 3
Joined: Tue Jul 17, 2018 4:07 pm

Re: Raw sensor access / CSI-2 receiver peripheral

Mon Jul 30, 2018 4:37 pm

Question relating to the process of adding sensor compatibility:

viewtopic.php?f=43&t=219263

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6879
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Mon Aug 20, 2018 5:07 pm

There appears to be a further issue with rawcam in the latest firmwares (since the July 30th release) where it is even more prone to getting buffers out of step. I am investigating.

If people want to play, then I have done a significant hack around with raspiraw to optionally run a very simple grey world AWB algorithm. See https://github.com/6by9/raspiraw/tree/temp_awb. It could do with some rough edges being cleaned up (almost no error handling!), but I'm tempted to merge it back into master anyway.
That branch also includes a fix for IMX219 having H & V flips swapped. I'm slightly surprised that hadn't been picked up before (or apologies if it had been reported and I totally missed it).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

VinoE
Posts: 2
Joined: Tue Aug 21, 2018 12:08 pm

Re: Raw sensor access / CSI-2 receiver peripheral

Thu Aug 23, 2018 10:04 am

Hello,

I'm just starting in RB Pi world and I would like some help with a project related to this thread's work. I have a Camera which sends its data (Pan or Bayer filtered) via 1 LVDS channel at 120MHz, it also has a Byte Strobe differential signal and the Clock.

I have a RB Pi Zero and I don't know if I can use its Camera connector and CSI 2 or if I should use the GPIO. In either case, besides the LVDS to Sub LVDS or 3v3CMOS conversion, what other aspects should I consider to achieve the speed and communication I want?

Thank you for your help,

Eddy

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6879
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Thu Aug 23, 2018 10:16 am

VinoE wrote:
Thu Aug 23, 2018 10:04 am
I'm just starting in RB Pi world and I would like some help with a project related to this thread's work. I have a Camera which sends its data (Pan or Bayer filtered) via 1 LVDS channel at 120MHz, it also has a Byte Strobe differential signal and the Clock.

I have a RB Pi Zero and I don't know if I can use its Camera connector and CSI 2 or if I should use the GPIO. In either case, besides the LVDS to Sub LVDS or 3v3CMOS conversion, what other aspects should I consider to achieve the speed and communication I want?
Sorry, the Pi doesn't support LVDS, and GPIOs will be too slow.

There are bridge chips around that will convert from various interfaces to MIPI CSI2. A quick Google would provide you with hits such as
http://www.latticesemi.com/Products/Des ... ridge.aspx (FPGA solution, and Sony sub-LVDS to CSI2)
or https://e2e.ti.com/support/interface/hi ... 8/t/614881 suggesting the TI DS90UH947-Q1 feeding a DS90UH940-Q1,
or http://www.lontiumsemi.com/uploadfiles/ ... _Brief.pdf

You'll need to do your own research to find a conversion solution that matches your use case.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6879
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Thu Aug 23, 2018 10:46 am

For those seing issues with rawcam, I've put a new firmware at https://drive.google.com/file/d/1pfj5OY ... sp=sharing that I believe fixes the problem. Please test and report back. It should get merged into the main firmware releases in the next few days anyway.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

VinoE
Posts: 2
Joined: Tue Aug 21, 2018 12:08 pm

Re: Raw sensor access / CSI-2 receiver peripheral

Thu Aug 23, 2018 12:10 pm

6by9 wrote:
Thu Aug 23, 2018 10:16 am
VinoE wrote:
Thu Aug 23, 2018 10:04 am
I'm just starting in RB Pi world and I would like some help with a project related to this thread's work. I have a Camera which sends its data (Pan or Bayer filtered) via 1 LVDS channel at 120MHz, it also has a Byte Strobe differential signal and the Clock.

I have a RB Pi Zero and I don't know if I can use its Camera connector and CSI 2 or if I should use the GPIO. In either case, besides the LVDS to Sub LVDS or 3v3CMOS conversion, what other aspects should I consider to achieve the speed and communication I want?
Sorry, the Pi doesn't support LVDS, and GPIOs will be too slow.

There are bridge chips around that will convert from various interfaces to MIPI CSI2. A quick Google would provide you with hits such as
http://www.latticesemi.com/Products/Des ... ridge.aspx (FPGA solution, and Sony sub-LVDS to CSI2)
or https://e2e.ti.com/support/interface/hi ... 8/t/614881 suggesting the TI DS90UH947-Q1 feeding a DS90UH940-Q1,
or http://www.lontiumsemi.com/uploadfiles/ ... _Brief.pdf

You'll need to do your own research to find a conversion solution that matches your use case.
Thanks for your repply 6by9. One last question, after the conversion to CSI2, what do I need to do in order to access these port in the Raspberry? As far as I read, this port only supports the Raspberry Camera Module.

Thank you,

Eddy

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6879
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Thu Aug 23, 2018 12:21 pm

VinoE wrote:
Thu Aug 23, 2018 12:10 pm
Thanks for your repply 6by9. One last question, after the conversion to CSI2, what do I need to do in order to access these port in the Raspberry? As far as I read, this port only supports the Raspberry Camera Module.
You have two choices, both of them requiring some effort from you.

First is that there is a V4L2 driver for the CSI2 peripheral. You need to connect it to a sensor driver, and it then delivers frames back via V4L2.
This is the preferred route.

Second there is the firmware based driver being discussed on this thread - a MMAL component called rawcam.
raspiraw is an example app that uses that component. All setup of the sensor is down to the application. raspiraw has example register commands to configure either version of the Pi camera (and an analogue video chip). You need to add the relevant config to set up your sensor.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
HermannSW
Posts: 1235
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Raw sensor access / CSI-2 receiver peripheral

Fri Oct 05, 2018 9:55 pm

I did not do much high framerate capturing lately. Only simple 180fps raspivid videos of payload dropped from drone.

I got asked whether Pi camera can do 2000x20@200fps, the 640xH high framerate modes were not sufficient horizontal resolution for scanner camera application.

I did test with v2 camera today, started with 3280x2464 tool and then: half vertical resolution, double framerate, repeat.
I ended up at 998fps for 3280x32, but a frame skip rate that I did not like.
I reduced to framerate with <1% frame skips, and that was 720fps.
I did same for 3280x64 and ended up with 500fps.

Summary:
v2 camera can do 3280x32@720fps and 3280x64@500fps with frame skip rate less than 1% (tools attached).
Similar to be expected for v1 camera (a little more than half the corresponding v2 framerate).
Attachments
3280fast.zip
(1.19 KiB) Downloaded 66 times
bookmark list: https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://github.com/Hermann-SW/fork-raspiraw
https://github.com/Hermann-SW/userland
https://twitter.com/HermannSW

Mike_green
Posts: 8
Joined: Wed Oct 17, 2018 6:36 am

Re: Raw sensor access / CSI-2 receiver peripheral

Mon Oct 22, 2018 9:00 am

Hi there! Here is our post about AR0144 sensor. Could you give us some tips?
viewtopic.php?f=43&t=225272

User avatar
HermannSW
Posts: 1235
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Raw sensor access / CSI-2 receiver peripheral

Sat Jan 05, 2019 6:22 pm

I saw that you got help on the sensor from 6by9 in the other thread.

Just wanted to report that for line scanner type applications v2 camera recording based on mode1 (1920x1080) can capture 1920x39@998fps with 1% frame skip rate. I looked into this only because mode1 is one of the no binning modes. Similar to the 640xH modes being capped at 1007fps, 1920xH modes are capped at 998fps. In the middle of the image you can see 2 of the 6 Arduino Due ISP pins, left and right from the the female headers:
Image
Attachments
1920x39.zip
(616 Bytes) Downloaded 31 times
bookmark list: https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://github.com/Hermann-SW/fork-raspiraw
https://github.com/Hermann-SW/userland
https://twitter.com/HermannSW

rcasiodu
Posts: 8
Joined: Tue May 28, 2019 2:28 am

Re: Raw sensor access / CSI-2 receiver peripheral

Thu May 30, 2019 9:22 am

6by9 wrote:
Thu Aug 23, 2018 12:21 pm
VinoE wrote:
Thu Aug 23, 2018 12:10 pm
Thanks for your repply 6by9. One last question, after the conversion to CSI2, what do I need to do in order to access these port in the Raspberry? As far as I read, this port only supports the Raspberry Camera Module.
You have two choices, both of them requiring some effort from you.

First is that there is a V4L2 driver for the CSI2 peripheral. You need to connect it to a sensor driver, and it then delivers frames back via V4L2.
This is the preferred route.

Second there is the firmware based driver being discussed on this thread - a MMAL component called rawcam.
raspiraw is an example app that uses that component. All setup of the sensor is down to the application. raspiraw has example register commands to configure either version of the Pi camera (and an analogue video chip). You need to add the relevant config to set up your sensor.
Hi 6by9, what's the main different of the two methods you mentioned?
In the first method, can you give me an example of drive file? Could I just modify the ov5647.c(/drivers/media/i2c/ov5647.c) to other sensor? How to use the v4l2 driver to get raw data from sensor? Is it possible to transfer raw10/raw12 to yuv data?(use arm core?)

Do the "vc.ril.isp" use the same pipeline of the isp in bcm283x for officially supported camera(ov5647 & imx219)? Or it just use the GPU core of bcm283x through MMAL?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6879
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Thu May 30, 2019 9:46 am

rcasiodu wrote:
Thu May 30, 2019 9:22 am
Hi 6by9, what's the main different of the two methods you mentioned?
The main ones:
- Licence. Any kernel driver almost has to be GPLv2 and therefore open source. Check with your sensor supplier that they are happy for the register set to be released. Copying raspiraw means that the register set can be hidden in your userspace app which can be under any licence you fancy (including closed). It's all a little silly as it is trivial to put an I2C analyser on the lines to the camera module, run your app, and capture the relevant commands.
- Framework. V4L2 is the Linux standard APIs and is therefore portable to other platforms. MMAL is specific to the Pi.
- Support. I'm less inclined to fix any issues discovered in rawcam than in V4L2. V4L2 is seen as the way forward.
rcasiodu wrote:In the first method, can you give me an example of drive file? Could I just modify the ov5647.c(/drivers/media/i2c/ov5647.c) to other sensor?
/drivers/media/i2c/ov5647.c is a relatively basic driver, but would work. imx258.c is a little more comprehensive, but lacks the dt/fwnode configuration and regulator control (not essential if you set the power control GPIO some other way). I believe ov5640.c is a reasonable example of that.
rcasiodu wrote:How to use the v4l2 driver to get raw data from sensor? Is it possible to transfer raw10/raw12 to yuv data?(use arm core?)
You use the V4L2 API to get raw frames out.
"v4l2-ctl --stream--map=3 --stream-to=foo.raw --stream-count=100" would be a simple existing tool.
https://linuxtv.org/downloads/v4l-dvb-a ... ure.c.html is the standard example for grabbing frames. Do what you want in process_image(). You want to be using the MMAP method, not read (userptr isn't supported).

The driver delivers the raw data, so that is whatever the sensor produces. You could try converting Bayer to YUV on the ARM, but you get a LOT of data, and the image processing required is not exactly lightweight.
Raw10/12 are a pain to unpack.There is an option to expand the raw10/12 to 16bpp (lowest bits used), but it's not exposed at the moment. unicam_set_packing_config does the relevant config, but currently mbus_depth will always equal v4l2_depth.

You can link V4L2 with MMAL. My fork of yavta (Yet Another V4L2 Test Application - https://github.com/6by9/yavta) does that, and uses dmabufs so there is no copying required of the data between the two subsystems.
Similarly in the 4.19 kernel there is now a V4L2 wrapper around the ISP which recent versions of GStreamer should talk to as v4l2videoconvert.

There is some work going on with http://libcamera.org/ to produce a standardised Linux framework for complex camera subsystems too.
rcasiodu wrote:Do the "vc.ril.isp" use the same pipeline of the isp in bcm283x for officially supported camera(ov5647 & imx219)? Or it just use the GPU core of bcm283x through MMAL?
Yes, it is the same hardware block, but with none of the control loops (AE/AGC/AWB/Len shading) running on the data.
On my list of tasks to support the libcamera stuff is to expose the statistics from the ISP so that those algorithms can be run more easily from userspace.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

rcasiodu
Posts: 8
Joined: Tue May 28, 2019 2:28 am

Re: Raw sensor access / CSI-2 receiver peripheral

Thu May 30, 2019 12:18 pm

Thanks 6by9, your answer are very helpfull. Now i understand the different and get more confidence about what i'm trying to do.

bhjel
Posts: 14
Joined: Tue Jan 01, 2019 2:00 am

Re: Raw sensor access / CSI-2 receiver peripheral

Sat Jun 01, 2019 10:39 am

Does GPU firmware limit the V1 cam to 90fps at 320x240 despite ov5647 sensor supporting 120 at 320x240? I have modified the kernel module to bump the cap to 120, but I can only achieve 120fps with the v2 camera.

User avatar
HermannSW
Posts: 1235
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Raw sensor access / CSI-2 receiver peripheral

Sat Jun 01, 2019 9:12 pm

bhjel wrote:
Sat Jun 01, 2019 10:39 am
Does GPU firmware limit the V1 cam to 90fps at 320x240 despite ov5647 sensor supporting 120 at 320x240?
Yes.
I have modified the kernel module to bump the cap to 120, but I can only achieve 120fps with the v2 camera.
What kernel module?
It is not raspivid open source userspace program that caps the framerate at 90fps, but closed source GPU code.

In my userland fork I did I2C command injection for framerates >90fps for both, v1 and v2 camera.
I proved that v2 camera can do up to 200fps videos and the cap at 120fps at that time was too conservative.
After that, 6by9 did bump maximal framerate for v2 camera mode 7 only to 200fps in closed source GPU.
Please see remarks below v2 camera mode table in documentation on framerates above 120fps:
https://www.raspberrypi.org/documentati ... /camera.md

Here you can see my code that allows for (arbitrary) overwrite of framerate, allowing to bypass GPU restriction:
https://github.com/Hermann-SW/userland/ ... id.c#L1682

v1 camera with framerate 90fps runs near bandwidth limit already for mode 7 :-(
You can get valid video for "-fps 95", wrong color video for "-fps 97" and you will get no video for framerate >97fps.


That is for mode 7. While there is no 320x240 mode, I made one for raspiraw and that showed that v1 camera can do 320x240@180fps (even 640x240@180fps), and raspiraw can even do 640x240@360fps with v2 camera. That mode is not that interesting though as does not work above 202fps because of a known GPU bug with won't fix decision:
https://github.com/6by9/userland/issues/14


A 320x240 mode 8 would make sense for v1 camera only, since v2 can do 640x480@180fps. Because of making sense for v1 camera only for raspivid, I doubt that 6by9 would consider adding that mode 8 for raspivid(yuv)? and raspistill.

Here is v1 camera raspiraw tool "640x240" that does capture with 180fps:
https://github.com/6by9/raspiraw/blob/m ... ls/640x240
bookmark list: https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://github.com/Hermann-SW/fork-raspiraw
https://github.com/Hermann-SW/userland
https://twitter.com/HermannSW

bhjel
Posts: 14
Joined: Tue Jan 01, 2019 2:00 am

Re: Raw sensor access / CSI-2 receiver peripheral

Sat Jun 01, 2019 9:35 pm

Thank you for the detailed post. Currently studying your changes and 6by9's work. I am using the V4l2 driver for my application which has an additional 90fps cap that limited even the v2 sensor before my change.:

"#define FPS_MAX 90"
https://github.com/raspberrypi/linux/bl ... 5-camera.c

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6879
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Sun Jun 02, 2019 6:46 am

bhjel wrote:
Sat Jun 01, 2019 9:35 pm
Thank you for the detailed post. Currently studying your changes and 6by9's work. I am using the V4l2 driver for my application which has an additional 90fps cap that limited even the v2 sensor before my change.:

"#define FPS_MAX 90"
https://github.com/raspberrypi/linux/bl ... 5-camera.c
This is really the wrong thread to be discussing anything to do with the bcm2835-v4l2 driver as that is sitting on top of the full camera stack, not rawcam and manually programming register sets.

I have had a reminder open for a while to try and make the max frame rate from V4L2 vary based on sensor, but I haven't got around to fixing it. https://github.com/raspberrypi/linux/issues/1774

In theory a QVGA 180fps mode could be added for the V1 sensor (assuming a register set from Omnivision actually worked), but the use cases are so limited that it isn't a priority. We haven't sold the V1 sensor since 2015 and I don't recall any requests for it until now.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Return to “Camera board”