mneg
Posts: 7
Joined: Thu Nov 29, 2018 9:21 am

Spatial drift on NoIR Camera v2

Thu Nov 29, 2018 10:09 am

I am testing the cam for a high-sensitivity application and retrieve unencoded video frames via raspividyuv (mode 4, luma only, downscaling to 416 x 320, low fps). Evaluation of the data revealed ugly spatial drifts of about 1 to 3 pixels (see graph).
spatial_drift.png
spatial_drift.png (11.54 KiB) Viewed 1659 times

First I thought thermal expansion of the cam holder or similar was responsible, however, different setups and repeated measurements always resulted in similar shift behaviour. The shifts are spatially irregular, i.e. the images are not alignable by global shift correction. Over time, the shift magnitude shows sudden hops which speaks against mechanical causes.

Could it be some Bayer denoising or demosaic-filter in the ISP pipeline introducing those shifts? Or could it even be deliberate noise inserted on a low level to prevent military use of the consumer camera?

The exact call is:
raspividyuv -set -sh 0 -co 0 -ex auto -fli off -awb off -awbg 1.27,1.30 -mm average -drc off -y -o video.y -t 120000 -md 4 -w 416 -h 320 -fps 2 --verbose

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6660
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Spatial drift on NoIR Camera v2

Thu Nov 29, 2018 11:23 am

I've reread that post three times now and I'm still no clearer as to what you're measuring. Some more detail and background please.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

mneg
Posts: 7
Joined: Thu Nov 29, 2018 9:21 am

Re: Spatial drift on NoIR Camera v2

Thu Nov 29, 2018 3:11 pm

6by9 wrote: I've reread that post three times now and I'm still no clearer as to what you're measuring. Some more detail and background please.
Sorry!

Setup: fixed camera, fixed scenery, nothing moves. Basically a noise characterization over several hundred frames. I cut out the initial frames containing the exposure adaption. With "spatial drifts" I mean displacements of image features rather than intensity drifts (which could simply be due to temperature variation of the CMOS chip).
The following image shows the first frame of a scene (left) and the difference of the first frame and the last frame (right). It's the same data used in the previous graph which shows the amplitude of the displacement.
spatial_drift_image.png
spatial_drift_image.png (53.79 KiB) Viewed 1617 times

The easy answer would be: the objects are moving. But they don't. I made several experiments, all showing similar curves for the displacement (slopes, hops). Now I wonder what the origin of those drifts could be and if there is a way to work around them.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 22106
Joined: Sat Jul 30, 2011 7:41 pm

Re: Spatial drift on NoIR Camera v2

Thu Nov 29, 2018 4:36 pm

Are you positive this isn't mechnical? The slightest of slight taps on my desk can produce shifts on the sensor of more than a pixel. Even cars going by outside have the chance of creating enough vibrations to show a shift, especially if the target is some distance away.

We do not put in anything deliberate that could do something like this.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6660
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Spatial drift on NoIR Camera v2

Thu Nov 29, 2018 5:05 pm

Simple answer would be to take raspiraw, capture a sequence of raw frames, process them through dcraw (either on x86 or Pi) and compare those output frames. That takes the ISP out of the loop.

There is an image stabilisation algorithm built in, but it is disabled by default.
There should be nothing altering the image cropping between frames unless you tell it to change the zoom or crop rectangle.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

mneg
Posts: 7
Joined: Thu Nov 29, 2018 9:21 am

Re: Spatial drift on NoIR Camera v2

Thu Nov 29, 2018 6:07 pm

jamesh wrote:
Thu Nov 29, 2018 4:36 pm
Are you positive this isn't mechnical?

Yes, quite. I also was convinced that it is mechanical. However, completely different mechanical setups resulted in very similar curves for the displacement (spatial_drift.png). Taps at the desk or larger scale accustic waves would leave at least different time-fingerprints for different setups. And the setup shown is only a few 10 cm from the camera (I had the same amount of shifted pixels with 4 meter offset).
6by9 wrote: Simple answer would be to take raspiraw

I just tried and didn't get raspiraw to run on my RPi 3B+ because it doesn't find the cam on I2C bus:

Code: Select all

raspiraw -md 7 -t 1000 -ts tstamps.csv -hd0 hd0.32k -h 64 --vinc 1F --fps 10 -c 0 -sr 2 -o ./out.%04d.raw
Using i2C device /dev/i2c-0
RaspiRaw: Probing sensor ov5647 on addr 36
RaspiRaw: Probing sensor imx219 on addr 10
RaspiRaw: Probing sensor adv7282 on addr 21
RaspiRaw: No sensor found. Aborting

and before:

Code: Select all

./camera_i2c
setting GPIO for board revsion: a020d3
Raspberry Pi3/Pi3+
Set state of 133 to 1
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
70: -- -- -- -- -- -- -- --                         

Any hint?
6by9 wrote: There should be nothing altering the image cropping between frames unless you tell it to change the zoom or crop rectangle.

The creepy thing is that it's no uniform shift which would result from linear change of crop or zoom.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6660
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Spatial drift on NoIR Camera v2

Fri Nov 30, 2018 9:55 am

mneg wrote:
Thu Nov 29, 2018 6:07 pm

Code: Select all

./camera_i2c
setting GPIO for board revsion: a020d3
Raspberry Pi3/Pi3+
Set state of 133 to 1
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
70: -- -- -- -- -- -- -- --                         
Any hint?
Please provide the output from "raspi-gpio get" after you have run "./camera_i2c".
Try manually running "sudo i2cdetect -y 0" after "./camera_i2c". The camera shouldn't take more than a few milliseconds to power up, and being on address 0x10 there are 13 addresses to probe first (0x00-0x02 are reserved).
mneg wrote:
6by9 wrote: There should be nothing altering the image cropping between frames unless you tell it to change the zoom or crop rectangle.
The creepy thing is that it's no uniform shift which would result from linear change of crop or zoom.
Uninitialised variables can do very funny things. Either that or rounding errors from funny maths.
But still, neither of those should be happening.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

mneg
Posts: 7
Joined: Thu Nov 29, 2018 9:21 am

Re: Spatial drift on NoIR Camera v2

Fri Nov 30, 2018 10:51 am

After fresh reboot and running raspiraw/camera_i2c (same output as in last post):
raspi-gpio get
BANK0 (GPIO 0 to 27):
GPIO 0: level=1 fsel=0 func=INPUT
GPIO 1: level=1 fsel=0 func=INPUT
GPIO 2: level=1 fsel=4 alt=0 func=SDA1
GPIO 3: level=1 fsel=4 alt=0 func=SCL1
GPIO 4: level=0 fsel=0 func=INPUT
GPIO 5: level=1 fsel=0 func=INPUT
GPIO 6: level=1 fsel=0 func=INPUT
GPIO 7: level=1 fsel=1 func=OUTPUT
GPIO 8: level=1 fsel=1 func=OUTPUT
GPIO 9: level=0 fsel=4 alt=0 func=SPI0_MISO
GPIO 10: level=0 fsel=4 alt=0 func=SPI0_MOSI
GPIO 11: level=0 fsel=4 alt=0 func=SPI0_SCLK
GPIO 12: level=0 fsel=0 func=INPUT
GPIO 13: level=0 fsel=0 func=INPUT
GPIO 14: level=1 fsel=2 alt=5 func=TXD1
GPIO 15: level=1 fsel=2 alt=5 func=RXD1
GPIO 16: level=0 fsel=0 func=INPUT
GPIO 17: level=0 fsel=0 func=INPUT
GPIO 18: level=0 fsel=0 func=INPUT
GPIO 19: level=0 fsel=0 func=INPUT
GPIO 20: level=0 fsel=0 func=INPUT
GPIO 21: level=0 fsel=0 func=INPUT
GPIO 22: level=0 fsel=0 func=INPUT
GPIO 23: level=0 fsel=0 func=INPUT
GPIO 24: level=0 fsel=0 func=INPUT
GPIO 25: level=0 fsel=0 func=INPUT
GPIO 26: level=0 fsel=0 func=INPUT
GPIO 27: level=0 fsel=0 func=INPUT
BANK1 (GPIO 28 to 45):
GPIO 28: level=1 fsel=0 func=INPUT
GPIO 29: level=0 fsel=0 func=INPUT
GPIO 30: level=0 fsel=7 alt=3 func=CTS0
GPIO 31: level=0 fsel=7 alt=3 func=RTS0
GPIO 32: level=1 fsel=7 alt=3 func=TXD0
GPIO 33: level=1 fsel=7 alt=3 func=RXD0
GPIO 34: level=0 fsel=7 alt=3 func=SD1_CLK
GPIO 35: level=1 fsel=7 alt=3 func=SD1_CMD
GPIO 36: level=1 fsel=7 alt=3 func=SD1_DAT0
GPIO 37: level=1 fsel=7 alt=3 func=SD1_DAT1
GPIO 38: level=1 fsel=7 alt=3 func=SD1_DAT2
GPIO 39: level=1 fsel=7 alt=3 func=SD1_DAT3
GPIO 40: level=0 fsel=4 alt=0 func=PWM0
GPIO 41: level=0 fsel=4 alt=0 func=PWM1
GPIO 42: level=0 fsel=4 alt=0 func=GPCLK1
GPIO 43: level=1 fsel=4 alt=0 func=GPCLK2
GPIO 44: level=1 fsel=5 alt=1 func=SDA0
GPIO 45: level=1 fsel=5 alt=1 func=SCL0
BANK2 (GPIO 46 to 53):
GPIO 46: level=1 fsel=0 func=INPUT
GPIO 47: level=1 fsel=1 func=OUTPUT
GPIO 48: level=0 fsel=4 alt=0 func=SD0_CLK
GPIO 49: level=1 fsel=4 alt=0 func=SD0_CMD
GPIO 50: level=1 fsel=4 alt=0 func=SD0_DAT0
GPIO 51: level=1 fsel=4 alt=0 func=SD0_DAT1
GPIO 52: level=1 fsel=4 alt=0 func=SD0_DAT2
GPIO 53: level=1 fsel=4 alt=0 func=SD0_DAT3

and
sudo i2cdetect -y 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: -- -- -- -- -- -- -- --

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6660
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Spatial drift on NoIR Camera v2

Fri Nov 30, 2018 11:50 am

I2C is muxed correctly based on that raspi-gpio output.
I'm assuming this camera is a genuine Raspberry Pi camera, and directly connected (not via one of the funny mux boards that are available).

You can try adding "./rpi3-gpiovirtbuf s 134 1" after "./camera_i2c" and before i2cdetect. That would turn the LED on on the original V1 boards, but should do nothing on a V2.
Alternatively run "./rpi3-gpiovirtbuf s 133" to confirm the state of the camera shutdown line. It might be worth checking the state of 133 and 134 whilst running raspivid or similar, as that would confirm what the firmware is setting up (although I can't see it being any different).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

mneg
Posts: 7
Joined: Thu Nov 29, 2018 9:21 am

Re: Spatial drift on NoIR Camera v2

Fri Nov 30, 2018 12:10 pm

6by9 wrote: I'm assuming this camera is a genuine Raspberry Pi camera, and directly connected (not via one of the funny mux boards that are available).

Aye, it's a genuine NoIR v2 module.
6by9 wrote: You can try adding "./rpi3-gpiovirtbuf s 134 1" after "./camera_i2c" and before i2cdetect. That would turn the LED on on the original V1 boards, but should do nothing on a V2.

Didn't see any LED lighting up on the camera board after applying the commands.
6by9 wrote: Alternatively run "./rpi3-gpiovirtbuf s 133" to confirm the state of the camera shutdown line. It might be worth checking the state of 133 and 134 whilst running raspivid or similar, as that would confirm what the firmware is setting up (although I can't see it being any different).

Affirmative. Both are 1 when raspividyuv is running, otherwise both are zero.

Maybe I should mention that /dev/i2c-0 was not there so I created a symbolic link from /dev/i2c-1. Seems as if /dev/i2c-0 is hardcoded in camera_i2c binary.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6660
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Spatial drift on NoIR Camera v2

Fri Nov 30, 2018 1:44 pm

mneg wrote:
Fri Nov 30, 2018 12:10 pm
Didn't see any LED lighting up on the camera board after applying the commands.
You won't - only the V1 had an LED.
mneg wrote:Maybe I should mention that /dev/i2c-0 was not there so I created a symbolic link from /dev/i2c-1. Seems as if /dev/i2c-0 is hardcoded in camera_i2c binary.
That would be it.
Add "dtparam=i2c_vc=on" to /boot/config.txt. i2c-0 and i2c-1 are different physical interfaces. i2c-0 is normally reserved for VideoCore (hence the vc designation) for talking to the cameras and display. i2c-1 is available for general use by the ARM.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

mneg
Posts: 7
Joined: Thu Nov 29, 2018 9:21 am

Re: Spatial drift on NoIR Camera v2

Fri Nov 30, 2018 4:21 pm

6by9 wrote: That would be it.
It was indeed. Capturing raw frames now works up to .ppm conversion using your latest dcraw.

However, for the experiment I would like to save only the sum of all RGB-elements of a pixel (i.e. Bayer-ignorant binning) of a subregion of the image to 16 bit values because the mode-4 frames are much too huge. I am thinking of replacing "fwrite(buffer->user_data, buffer->length, 1, file)" in raspiraw.c with a binary dump of my averaged values. For averaging I would use your get_pixel() and get_channel() functions. Would you consider that approach wise or am I overseeing existing facilities?

Is there a special reason you use -O0 optimization in raspiraw?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6660
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Spatial drift on NoIR Camera v2

Fri Nov 30, 2018 5:06 pm

mneg wrote:
Fri Nov 30, 2018 4:21 pm
However, for the experiment I would like to save only the sum of all RGB-elements of a pixel (i.e. Bayer-ignorant binning) of a subregion of the image to 16 bit values because the mode-4 frames are much too huge. I am thinking of replacing "fwrite(buffer->user_data, buffer->length, 1, file)" in raspiraw.c with a binary dump of my averaged values. For averaging I would use your get_pixel() and get_channel() functions. Would you consider that approach wise or am I overseeing existing facilities?
You'll get some very weird effects due to the variation in sensitivities of the 4 colour channels, and dcraw won't be able to process it as the header will be invalid (although you could attempt to correct).
raspiraw really is leaving everything up to you. I can't predict what people want to do with it, so you're pretty much on your own. I was suggesting you use it for investigating spatial drift only, not general use.
mneg wrote:Is there a special reason you use -O0 optimization in raspiraw?
Only because it makes life easier when debugging in gdb. Very little of raspiraw is time critical, therefore I don't care about optimisations.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

mneg
Posts: 7
Joined: Thu Nov 29, 2018 9:21 am

Re: Spatial drift on NoIR Camera v2

Sat Dec 01, 2018 10:38 am

Further results of the spatial drift analysis:

Setup: high-detail object in 30 cm distance to camera with one source of illumination. All objects are on the same platform.
Measurement: 120 seconds raw data recording (raspiraw) and YUV recording (raspividyuv). Different fps were used.

Left of each figure is the original frame #1. To the right is the difference of frame #1 and the frame @120 s.
The attachment ISP_img.jpg is no longer available
ISP_img.jpg
YUV recording
ISP_img.jpg (54.73 KiB) Viewed 1353 times

It looks like the drift is not due to the ISP. The magnitude of drift shows similar time-behaviour for both recording conditions:
raw_img.jpg
raw recording
raw_img.jpg (48.84 KiB) Viewed 1351 times

During AD-conversion the chip produces more heat. Could it be some thermal expansion of the lens or housing? And the hops from the previous analysis some mechanical stick-slip effect?
Attachments
drift.jpg
drift.jpg (16.23 KiB) Viewed 1351 times

HermannSW
Posts: 1210
Joined: Fri Jul 22, 2016 9:09 pm

Re: Spatial drift on NoIR Camera v2

Sun Dec 02, 2018 8:57 am

In my raspiraw recordings 1st frame is often not good, sometimes even the first few.
To avoid hunting for a ghost, I would take frame 5 to compare with later frames.
bookmark list: https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://github.com/Hermann-SW/fork-raspiraw
https://github.com/Hermann-SW/userland
https://twitter.com/HermannSW

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6660
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Spatial drift on NoIR Camera v2

Mon Dec 03, 2018 10:38 am

If the shift is observed in the raw Bayer that is coming from the sensor via raspiraw then there is very little we can do about it.
(Hermann's comment on the first frame not necessarily being reliable excepted).

The sensor does have an onboard temperature sensor, although it is very coarse ("The target range is -10 to 95 degrees in C, with +/- 5 degree in C deviation.").
Adding the code section

Code: Select all

@@ -1933,7 +1933,46 @@ int main(int argc, char** argv) {
 
        start_camera_streaming(sensor, sensor_mode);
 
-       vcos_sleep(cfg.timeout);
+       for (i = 0; i < cfg.timeout/1000; i++)
+       {
+               int fd;
+               vcos_sleep(1000);
+               fd = open(i2c_device_name, O_RDWR);
+               if (!fd)
+               {
+                       vcos_log_error("Couldn't open I2C device");
+                       continue;
+               }
+               if (ioctl(fd, I2C_SLAVE_FORCE, sensor->i2c_addr) < 0)
+               {
+                       vcos_log_error("Failed to set I2C address");
+                       close(fd);
+                       continue;
+               }
+
+               if (sensor->i2c_addressing == 2)
+               {
+                       unsigned char msg[4] = {0x0140>>8, 0x0140 & 0xff, 0x80};
+                       int len = 3;
+
+                       if (write(fd, msg, len) != len)
+                       {
+                               vcos_log_error("Failed to write register 0x0140");
+                       }
+               }
+               vcos_sleep(5);
+               {
+                       uint8_t reg;
+                       float temp;
+                       if(i2c_rd(fd, sensor->i2c_addr, 0x0140, (uint8_t*)&reg, 1, sensor))
+                       {
+                               vcos_log_error("Failed to read register 0x0140");
+                       }
+                       temp = ((reg&0x7f)*5.0/6.0) - 10;
+                       vcos_log_error("temp sensor reports %02X, %f degC", reg, temp);
+               }
+               close(fd);
+       }
 
        stop_camera_streaming(sensor);
so that it is polled every second whilst the sensor is running seems to show relatively little variation on my sensor. The full range of values do appear to be returned, so it's probably just the accuracy of the recorded values that is 5degC.

Sorry, it seems to be that it is what it is.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Return to “Camera board”