Raspiraw problems with NoIR V2

[RDX705]

I installed Raspiraw on my RPI 3B+ and experienced some problems with high-speed video capture:

1. No matter what resolution and FPS I set in ./raspiraw command, FPS never exceeds 243 (randomly fluctuates between 235-242), with values lower than 243 setting is fine. Note: I check for fps both via frames recorded in timestamp file and "ls -l /dev/shm/out.*.raw | wc --lines" command.

2. Captured frames are either pitch black or a seemingly random amalgamation of colored pixels even after conversion (raw>ppm>png).

3. All recording executables in "tools" section such as "640x480" and similar don't run properly, displaying errors that /dev/shm/out.*.raw and timestamp file were not found.

What is possible reason for those problems?

[HermannSW]

Which repo did you install?

[RDX705]

https://github.com/6by9/raspiraw

[HermannSW]

Ok, that explains it.
Only six of the tools in tools folder work for v2 camera, all other tools are for v1 camera, see the comments:
https://github.com/6by9/raspiraw/tree/master/tools

The same is true for my working fork, it has some code changes more, but the v2 tools are not merged:
https://github.com/Hermann-SW/fork-rasp ... ster/tools

I did post separate tools/v1 and tools/v2 directories in a code drop last May:
https://www.raspberrypi.org/forums/view ... 2#p1320044

tools/v2 directory contains more than 30 v2 camera raspiraw tools, including all made for below height vs. framerate diagram for 640xH tools. There is no limit at 200 something fps for v2 camera, you can do up to 1007fps with 640x75 or 640x150_s tools! And the tools tell you on framerate achieved, as well on frame skip count. This is one of my favorite 640x75@1007fps videos of a closing mouse trap (played at 1fps, closing completes in 0.01s). You can see the spark in front when closing mouse trap bar hits black piezo igniter at the end:




P.S:
I did capture a single frame (but 1920x1080!) of closing mouse trap much sharper and without rolling shutter effect using Arducam ov5647 camera (v1 camera with exposed FREX and STROBE lines) using strobe lens of 11µs:
https://www.raspberrypi.org/forums/view ... 1#p1451651

[RDX705]

Thanks. Will try to use V2 tools and see if it works.

P.S. Mouse trap looks amazing! What else did you record with hi-speed camera?

[HermannSW]

> P.S. Mouse trap looks amazing! What else did you record with hi-speed camera?
>
Bookmark list from my signature contains more examples.
The Raspiraw section there contains links to relevant postings/threads:
https://stamm-wilbrandt.de/en/Raspberry ... l#raspiraw
This is another mouse trap video, "only" 350fps, but 640x128 with v1 camera (played at 25fps, 14 times slower than real):



These are other high speed objects I looked at, missing is mini done propeller with 124m/s(!) blade tip radial speed:
https://www.raspberrypi.org/forums/view ... 7#p1443533



Speeds above say 30m/s are bad for v2 camera, this is biggest rolling shutter effect I have ever captured in video or photo (propeller running at "only" 46m/s blade tip radial speed):
https://www.raspberrypi.org/forums/view ... 6#p1449729

[RDX705]

Tried v2 tools, same problem as before - cut:tstamps.csv: no such file or directory, ls: cannot access '/dev/shm/out.'.raw": no such file or directory. It also has some kind of syntax error on line 14 - operand expected after error token.

However, raspiraw is working fine now, getting correct amount of frames instead of being stuck around 240. BTW, what values should I pass for "-r" argument in command if my sensor is imx219? Currently its "380A,0040;3802,78;3806,0603" but AFAIK its for different sensor.

[RDX705]

Captured frames are extremely bright now - is it because of incorrect exposure time?

[HermannSW]

However, raspiraw is working fine now, getting correct amount of frames instead of being stuck around 240. BTW, what values should I pass for "-r" argument in command if my sensor is imx219? Currently its "380A,0040;3802,78;3806,0603" but AFAIK its for different sensor.

Please make sure you run the correct tools, the register settings don't occur in any of the drop's v1 or v2 directory tools:

Code: Select all

$ grep "\-r" raspiraw.drop.5_26_18/v?/*
$ 

Captured frames are extremely bright now - is it because of incorrect exposure time?

Please light the scene with bright light. For 665fps v1 or 1007fps v2 camera captures I used 1000lm Ikea lamp.

The deal with Raspverry camera high framerate recording is basically this:

• halve vertical height of recording and get roughly double framerate (unfortunately there is a hard limit at 1007fps for v2)
• for doubling framerate of recording you need to increase scene lighting

[RDX705]

Is it possible to record at 250-350 fps without bright light? I wanted to film birds outside with hi-speed camera, will natural lightning suffice?

[HermannSW]

Is it possible to record at 250-350 fps without bright light? I wanted to film birds outside with hi-speed camera, will natural lightning suffice?

Sorry, I was not able to do high framerate recording with raspiraw without bright light.

Only once I was able to do 1007fps raspiraw recording through fast running train window, but the sun in Rhine valley was very bright that day:
https://www.raspberrypi.org/forums/view ... 8#p1433868



In case 180fps are good enough for you, I would go with v2 camera and raspivid "--fps 180".

https://www.raspberrypi.org/forums/view ... 7#p1285069
This is 56MB animated .gif from the video, played at 25fps, 7.2 times slower than real:
https://stamm-wilbrandt.de/en/forum/bridge.anim.gif



6by9 asked me to always state the restrictions in case going above 120fps with v2 camera raspivid per the documentation.
https://www.raspberrypi.org/documentati ... /camera.md

7 640x480 4:3 40-200fps¹ Partial 2x2

¹For frame rates over 120fps, it is necessary to turn off automatic exposure and gain control using -ex off. Doing so should achieve the higher frame rates, but exposure time and gains will need to be set to fixed values supplied by the user.

Although I took numerous fine 180fps videos just without that steps.
You can use raspivid with 640x480 mode 7 up to "--fps 150" without frameskips.
You can use it up to "--fps 170" without frame skips in case of adding "-ex off" per the documentation.
https://www.raspberrypi.org/forums/view ... 3&t=231596

The advantage of running v2 raspivid at 150/170fps over raspiraw is that you have the complete GPU processing of raspivid and generally nicer looking videos than with raspiraw. In case more than 202fps are needed you have to go with raspiraw though because Raspberry GPU locks up, and that will not be fixed:
https://github.com/6by9/userland/issues/14

[RDX705]

OK, thanks.

By the way, what does "unexpected and of file" mean in raw2ogg2anim? When I get this warning, half of the screen in .ogg video becomes kind of underprocessed.

[HermannSW]

For normal tools like 640x128 you just call "raw2ogg2anim videoname first last fps" (targetFps, first and last frame)
For tools like 640x128_s you call "raw2ogg2anim videoname first last fps d".
For tools like 1296x720_S you call "raw2ogg2anim videoname first last fps dd".

[RDX705]

Hello again. I have some questions on Raspiraw.

1) Is it possible to capture greyscale/binary images with Raspiraw?
2) Is it possible to output ppm/png files?

P.S. I tested it outside during bright sunny day in 640x240, and resulting frames are of quite high quality. This may be because NoIR doesn't have IR filter however.

[6by9]

1) Is it possible to capture greyscale/binary images with Raspiraw?

Not directly. Predominantly raspiraw is capturing the raw image data off the sensor with no processing. As with most CMOS colour sensors, the Pi camera modules are not a complete RGB array but have a Bayer colour filter - https://en.wikipedia.org/wiki/Bayer_filter. An image processing technique called demosaicing is used to convert that pattern into a complete RGB (or YUV) image.

raspiraw does now have an option for saving the image data after it has been through the ISP (Image Sensor Pipeline. The hardware block for image processing on the Pi). Currently that is always a YUV frame, and the Y is the luma component which equates to 8bpp greyscale. If you saved just the first width*height bytes of the buffer instead of all of it, then you'd have an 8bpp greyscale image. Without white balance the composition may be off though.

2) Is it possible to output ppm/png files?

Again, not directly.
IIRC PPM can be created with a simple header to the raw data.
PNG is generally losslessly compressed, so that would require additional processing.
I don't know the details on how either represent greyscale images.

[HermannSW]

1) Is it possible to capture greyscale/binary images with Raspiraw?

The bg/Gr pixels of raw Bayer 2x2 tiles are bightest at left bottom green G pixel.
In this posting you can see how to intercept raspiraw and reduce 640x480 raw Bayer to 320x240 grey frame on the fly:
https://www.raspberrypi.org/forums/view ... 1#p1231151

2) Is it possible to output ppm/png files?

In the same posting you can find my code to write .pgm (protable grey map) files. I used that to store the image after tilt calibration of the robot camera completed (for tilt calibration I did a conversion from grey8 to b&w for easier calibration motor control code):



P.S:
The caterpillar robot I used in that posting had maximal speed of 2.5m/s or 9km/h.
Today I ordered a second T101 tank robot chassis:
https://www.aliexpress.com/item/32813434124.html

While the gear motor delivered with T101 has 330rpm only, I do have same size 1500rpm gear motors already.
With wheel diameter of 65mm and 1500rpm maximal speed of 5.1m/s or more than 18km/h should be possible.

Until recently I planned to do video processing based on v1 camera with raspiraw high framerate processing.
Lately I got free samples of ov7251 Mipi monochrome global shutter camera from Lee, and will most likely go with that (no dealing with colors because images are monochrome already, and global shutter is likely advantageous for 5m/s moving robot, >100fps for 640x480, 200/400fps possible with smaller resolutions).

[RDX705]

I have seen your post regarding usage of raspiraw for robot guidance and decided to build a turret robot using Lego NXT motors and RPI, that will just spin in one place and "scan" surroundings with camera. I made an algorithm using findContours from OpenCV (it seems to be much faster than Canny) to detect any suspicious objects (lego block towers in this case) and display direction to them as output. Is it possible to direct Raspiraw output (grey frames) at another process (my algorithm in this case)? Maybe via simple udp sender-receiver (former in Raspiraw, latter - in algorithm) or via gstreamer pipeline?
I am concerned that if raspiraw will simply save frames somewhere and then algorithm will read and process them, latter will fall behind the former.

[HermannSW]

Is it possible to direct Raspiraw output (grey frames) at ... or via gstreamer pipeline?

Yes, see "gstreamer plugin dev / appsrc / raspiraw / Raspberry camera" thread:
https://www.raspberrypi.org/forums/view ... p?t=197124

Code: Select all

$ time raspiraw -md 7 -t 5000 -sr 1 -o "appsrc name=_ ! videoconvert ! fpsdisplaysink name=# video-sink='fakesink'" 2>err 1>out

I am concerned that if raspiraw will simply save frames somewhere and then algorithm will read and process them, latter will fall behind the former.

That can happen, but in the posting I pointed to the code was integrated into raspiraw. While that is no nice separation, it guarantees synchronous execution, each frame received from camera gets immediately processed and triggers stepper motor change if needed.


For the ov7251 Mipi camera acting on the frames can be easily done in video_callback() function:
https://github.com/ArduCAM/MIPI_Camera/ ... ideo.c#L10

In video.c that function gets called with variable size h264 encoded frames, but I know from a previous driver that a stream of (constant size) grey frames can be retrieved from the camera as well (at 135fps for 640x480). and that would be where I would add my robot video processing code (no need to store the frames). [last night I measured 132fps of h264 video.c with camera pointed to an Android stop watch with 1/100th second resolution by single frame stepping the video uploaded to youtube: https://www.youtube.com/watch?v=txGash5ll6A]

[RDX705]

Thanks. Where should I put the code to output black and white frames? In callback() method?

[HermannSW]

Yes, you can skip the state machine with stepper motor control. Do the output just after the grey8 to b&w conversion step. In case you skip the b&w conversion, the frames you would save grey8 .pgm files.

[RDX705]

I tried adding b/w conversion to the callback method, but I still get coloured images. What could be the reason for this?

Code: Select all

static void callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer)
{
	static int count = 0;
	vcos_log_error("Buffer %p returned, filled %d, timestamp %llu, flags %04X", buffer, buffer->length, buffer->pts, buffer->flags);
	if (running)
	{
		RASPIRAW_PARAMS_T *cfg = (RASPIRAW_PARAMS_T *)port->userdata;

		if (!(buffer->flags&MMAL_BUFFER_HEADER_FLAG_CODECSIDEINFO) &&
                    (((count++)%cfg->saverate)==0) && (count > 1))
		{
			// Save every Nth frame
			// SD card access is too slow to do much more.
			FILE *file;
			char *filename = NULL;
			if (create_filenames(&filename, cfg->output, count) == MMAL_SUCCESS)
			{
				
				file = fopen(filename, "wb");
				if (file)
				{
					if (cfg->ptso)  // make sure previous malloc() was successful
					{
						cfg->ptso->idx = count;
						cfg->ptso->pts = buffer->pts;
						cfg->ptso->nxt = malloc(sizeof(*cfg->ptso->nxt));
						cfg->ptso = cfg->ptso->nxt;
					}
					if (!cfg->write_empty)
					{
					
						if (cfg->write_header)
							fwrite(brcm_header, BRCM_RAW_HEADER_LENGTH, 1, file);
							
								/////
				int i,j,k,t;
                   unsigned char *p, *q;
                  // struct timeval start, end;
 
                  // gettimeofday(&start, NULL);
 
                   p = q = buffer->data;
 
                   for(i=1; i<480; i+=2) // calibration needs last three lines //480
                   {
                     p += 800;
  
                     for(j=0; j<640; j+=4, p+=5)        // convert raw10 to .pgm
                     {
                       k = (((int) p[0])<<2) + ((p[4]>>0)%0x03);
                       *q++ = (k>=50) ? 255 : 0;                    // b/w filter
 
                       k = (((int) p[2])<<2) + ((p[4]>>4)%0x03);
                       *q++ = (k>=50) ? 255 : 0;
                     }
                   }
				  
 
				//////
						fwrite(buffer->data, buffer->length, 1, file);
					}
					fclose(file);
				}
				
				free(filename);
			}
		}
		buffer->length = 0;
		mmal_port_send_buffer(port, buffer);
	}
	else
		mmal_buffer_header_release(buffer);
}

[RDX705]

Is it possible to output files without .32k header?

[HermannSW]

Is it possible to output files without .32k header?

Of course, from raspiraw documentation:
https://github.com/6by9/raspiraw

-hd0, --header0 : Sets filename to write the BRCM header to

All tools make use of that, eg.:
https://github.com/6by9/raspiraw/blob/m ... ls/640x128

In post processing raw2ogg2anim joins the one saved 32K block with each frame captured allowing dcraw to process:
https://github.com/6by9/raspiraw/blob/m ... w2ogg2anim

P.S:
This option writes (intentionally) empty frames to determine bounds on framerate maximal possible:

-emp, --empty : Write empty output files

[RDX705]

I probably phrased question the wrong way. I meant if it is possible to obtain .raw image that can be immediately converted by dcraw without the need of "cat hd0.32k"?

[HermannSW]

Yes, don't use "-hd0" option and the 32k block (needed by dcraw) will be part of each frame output of raspiraw. This does of course has a negative effect on maximally achievable framerate with raspiraw.