.PNG and .JPEG photo capture performance - GPU overclocking

[Eichhorn18]

Hi,

I have been trying to capture high quality images from the picamera rapidly for a while now. I have been using the picamera python library.

I recently did a comparison between some of the photo formats that can be taken.

JPEG images with quality=100 (minimum compression), captures images at a rate of 1.7 photos per second
The JPEG photos were approximately 5 MB

PNG images were captured at a rate of 0.15 photos per second
The PNG photos were 13 MB

The PNG photos are obviously better quality and so my problem was then how to increase the capture rate. My questions for anyone who knows is how the Raspberry Pi captures JPEG photos differently from PNG. Its clear that the data manipulation for PNG leaves the image uncompressed entirely, but the resolution is the same as the JPEG. So the reduction in speed is coming from either the way the image is encoded, and if not the encoding, then it must be coming from the size of the photo and the pipeline the data takes to be written to an SD card.

I'm therefore trying to determine if A) I should switch image formats if its known the PNG photos take a long time to create, or B) Can I overclock the pi GPU / memory to increase this capture speed.

If anyone has some ideas about this I would appreciate hearing any / all suggestions.
Thanks

[itsmedoofer]

Hi,

An off hand comment as I really am not an expert, but could you shoot in a "RAW" and post process the files later.

That would to a degree lessen the amount of processing, however you could run into issues shifting more data about, slow writes to SD for example.....

Just a thought.

[6by9]

Two differences:
1) There is no hardware acceleration for PNG, whilst there is for JPEG.
2) PNG encodes an RGB image. JPEG encodes a YUV image. The camera always produces YUV, so therefore is a conversion from YUV to RGB required, and again this is done in software.
Basically there is no easy way to accelerate this. Increasing the vpu clock should give you a linear increase in performance should you wish to go that route.
(There would be a way to offload the conversion to RGB to hardware if you really felt like messing with MMAL pipelines. It won't be easy through picamera)

I'm a little surprised there's that big a difference between JPEG with quality 100 and PNG. JPEG quality 100 will be throwing away very little of the high frequency data, therefore it should be pretty close. Can you quantify what you perceive as the difference in image quality?

[knute]

PNG is not a good format for photographs. PNG is better for sharp edged drawings and graphics, JPEG is the best format for a photograph. JPEG's one major drawback is that every time you save a JPEG photo you lose some data even at 100% compression.

[jamesh]

PNG is not a good format for photographs. PNG is better for sharp edged drawings and graphics, JPEG is the best format for a photograph. JPEG's one major drawback is that every time you save a JPEG photo you lose some data even at 100% compression.

I would actually say it's the other way round. png is a lossless format, jpeg is lossy.

[Eichhorn18]

Two differences:
1) There is no hardware acceleration for PNG, whilst there is for JPEG.
2) PNG encodes an RGB image. JPEG encodes a YUV image. The camera always produces YUV, so therefore is a conversion from YUV to RGB required, and again this is done in software.
Basically there is no easy way to accelerate this. Increasing the vpu clock should give you a linear increase in performance should you wish to go that route.
(There would be a way to offload the conversion to RGB to hardware if you really felt like messing with MMAL pipelines. It won't be easy through picamera)

I'm a little surprised there's that big a difference between JPEG with quality 100 and PNG. JPEG quality 100 will be throwing away very little of the high frequency data, therefore it should be pretty close. Can you quantify what you perceive as the difference in image quality?

Thanks for all the replies so far. An update on where I am now:

-I've tried to capture in using rgb directly into a numpy array, after which I save the array as an .npy file to the SD card (for post-processing into a bmp later). The size of the .npy file is 23 MB, and I have managed to capture this and write to the SD card in less than one second. I've posted a copy of the script at the end of this post.

-I can't immediately quantify the quality difference between the JPEG and PNG, but my application examines patterns of pixels to identify parts of an image. Any modification to the pixels is a degradation of the precision of repeat measurements made with photos of the same scene. Does that make any sense?

-In order to make this capture work at all I had to set GPU memory to 256 MB.

-I am now trying to determine why the code doesn't always complete properly (it takes the first few photos in the loop and then gets stuck with no error output--it just freezes). It is not consistently not working though--the capture program works perfectly sometimes, and other times gets hung up and I have to reboot. I can't determine what is the limiting process that could be hanging this up. I've tried to force_turbo=1, and given some over_voltage=2 to try and stabilise the program.

-a colleague has suggested that I may be putting the cpu and program into 'race conditions'.

Can anyone comment if my program is crashing because of a hardware limitation or interference of the software running and being interrupted by other system processes.

Code: Select all

#RGB capture to a numpy array, triggering from GPIO input
#Last updated August 24,2018
import time
import RPi.GPIO as GPIO
from time import sleep
import picamera
import numpy as np

def run():
    GPIO.setmode(GPIO.BOARD)
    GPIO.setup(19,GPIO.IN)
    capture_on = True
    
    frames = 4
    image_name = 'gandalf'
    filepath = '/home/pi/Desktop/ge267/WORKING/'+ image_name   
    
    #For capture with use_video_port, use resolution 1024x768
    with picamera.PiCamera() as camera:
        width = 3264 #requested output resolution
        height = 2464
        fwidth = (width + 31) // 32 * 32 #add buffer of array space required by capture to numpy
        fheight = (height + 15) // 16 * 16
        camera.resolution = (fwidth, fheight)
        camera.framerate = 30
        camera.iso = 100
        camera.rotation=180
        
        time.sleep(2) 

        i = 1
        output = np.empty((fheight, fwidth, 3), dtype=np.uint8) #create an empty numpy array
        while capture_on == True:
            start = time.time()
            camera.capture(output,format='rgb')
            np.save(filepath + '_frame%04d.npy' % i,output,allow_pickle='FALSE') #save the array as a binary file for post-processing
            finish = time.time()
            print('Frame number: frame%04d.npy' %i)
            print('Captured one frame in %.2f seconds' % (finish - start))
            
            if GPIO.input(19)==1: #escape options
                capture_on = False
            elif i >= frames:
                capture_on = False
            i += 1  
        camera.close()

[knute]

-I can't immediately quantify the quality difference between the JPEG and PNG, but my application examines patterns of pixels to identify parts of an image. Any modification to the pixels is a degradation of the precision of repeat measurements made with photos of the same scene. Does that make any sense?

Are you using a raspberry pi camera? It would be my guess that the camera is noisier than the distortion added by compressing to a 100% JPEG.

If you capture images only for a limited period of time, you might consider buffering them to a ramdisk which writes much faster than an SD card.

[Eichhorn18]

Are you using a raspberry pi camera? It would be my guess that the camera is noisier than the distortion added by compressing to a 100% JPEG.

If you capture images only for a limited period of time, you might consider buffering them to a ramdisk which writes much faster than an SD card.

Yes I am using the raspberry Pi Camera V2.1. I'm not sure that's true that the noise introduced by the camera sensor is more problematic than compression. I suppose it depends on the application and what you consider noise, and loss of data. Certainly for my application the smoothing of adjacent pixels by the JPEG compression is a reduction in image quality. The noise on the camera is non-controllable and so I think its not really a discussion point.

I considered capture to a RAM disk but with 23MB photos coming off the sensor, I would get only a few minutes of capture which is not meeting my need.

[6by9]

Do you need colour? Or more specifically do you need RGB?
23MB per frame is giving you nearly zero compression as the raw RGB at 24MB.

If you don't need colour then grab YUV frames and save only the luma (or process in the YUV colourspace) to give 8MB per frame. if you do then request RGB to get 24MB of data and save the overhead of trying to compress the data.

I'm not going to benchmark the encode in detail, but it seems to be taking several seconds before returning data which is unexpected. It appears to be the encode that is holding it all up as the preview has resumed and then we get a shed load of data transfer. When encoding a JPEG the preview is paused during the whole encode as it does the image incrementally.

If you use raspiyuv -rgb to grab the raw image, then I'm seeing the capture take < 0.5sec (I haven't timed it accurately) - that's more like what I'm expecting, and matches your capture to a numpy array.
There is no vector acceleration of PNG encoding on the GPU, therefore you may find it faster to do it on the ARM cores than asking the GPU to do it (250MHz processor vs 1.4GHz processor). That also allows the GPU to move on to capturing the next still for you.

[Eichhorn18]

Do you need colour? Or more specifically do you need RGB?
23MB per frame is giving you nearly zero compression as the raw RGB at 24MB.

What I really need is a .bmp with colour but capturing directly to a bmp takes too long. I capture as RGB into an array and then post-process it offline of the raspberry pi into a .bmp.

If you don't need colour then grab YUV frames and save only the luma (or process in the YUV colourspace) to give 8MB per frame. if you do then request RGB to get 24MB of data and save the overhead of trying to compress the data.

I tried capturing as YUV to the array but then had difficulty post-processing it, I think due to the padding at the beginning of the binary file. Can you elaborate on how much additional data is added to the YUV file in addition to the actual Y, U, and V, data?

I'm not going to benchmark the encode in detail, but it seems to be taking several seconds before returning data which is unexpected. It appears to be the encode that is holding it all up as the preview has resumed and then we get a shed load of data transfer. When encoding a JPEG the preview is paused during the whole encode as it does the image incrementally.

If you use raspiyuv -rgb to grab the raw image, then I'm seeing the capture take < 0.5sec (I haven't timed it accurately) - that's more like what I'm expecting, and matches your capture to a numpy array.
There is no vector acceleration of PNG encoding on the GPU, therefore you may find it faster to do it on the ARM cores than asking the GPU to do it (250MHz processor vs 1.4GHz processor). That also allows the GPU to move on to capturing the next still for you.

To be clear, when I'm using capture to an array (3264,2464,3) as RGB format--that step along is taking about 0.6 sec. I can do this indefintely, overwriting the variable in ram that is taking the rgb data. The issue occurs when I try to write the data to the SD card. This step in my code completes in an additional 0.1 to 0.2 sec and the loop to capture restarts, but then sometimes it gets hung up.

How would I attempt your last suggestion? Running it on the ARM core rather than the GPU?

[6by9]

Do you need colour? Or more specifically do you need RGB?
23MB per frame is giving you nearly zero compression as the raw RGB at 24MB.

What I really need is a .bmp with colour but capturing directly to a bmp takes too long. I capture as RGB into an array and then post-process it offline of the raspberry pi into a .bmp.

BMP is really bad as IIRC it encodes upside down, therefore the camera has to have fully finished before it can start encoding.

If you don't need colour then grab YUV frames and save only the luma (or process in the YUV colourspace) to give 8MB per frame. if you do then request RGB to get 24MB of data and save the overhead of trying to compress the data.

I tried capturing as YUV to the array but then had difficulty post-processing it, I think due to the padding at the beginning of the binary file. Can you elaborate on how much additional data is added to the YUV file in addition to the actual Y, U, and V, data?

There is no padding at the beginning of the file.
The stride of the image is rounded up to a multiple of 32, and the height to a multiple of 16, but 3264 x 2464 already meets those criteria.
You will get a 3264x2464 8bpp luma (Y) plane, followed by 1632x1232 8bpp of U, and then 1632x1232 8bpp of V.

I'm not going to benchmark the encode in detail, but it seems to be taking several seconds before returning data which is unexpected. It appears to be the encode that is holding it all up as the preview has resumed and then we get a shed load of data transfer. When encoding a JPEG the preview is paused during the whole encode as it does the image incrementally.

If you use raspiyuv -rgb to grab the raw image, then I'm seeing the capture take < 0.5sec (I haven't timed it accurately) - that's more like what I'm expecting, and matches your capture to a numpy array.
There is no vector acceleration of PNG encoding on the GPU, therefore you may find it faster to do it on the ARM cores than asking the GPU to do it (250MHz processor vs 1.4GHz processor). That also allows the GPU to move on to capturing the next still for you.

To be clear, when I'm using capture to an array (3264,2464,3) as RGB format--that step along is taking about 0.6 sec. I can do this indefintely, overwriting the variable in ram that is taking the rgb data. The issue occurs when I try to write the data to the SD card. This step in my code completes in an additional 0.1 to 0.2 sec and the loop to capture restarts, but then sometimes it gets hung up.

How would I attempt your last suggestion? Running it on the ARM core rather than the GPU?

http://www.libpng.org/pub/png/libpng.html
Having tried using imagemagik to convert a very low quality 5MP JPEG to PNG, that's taken 1.6s on a Pi3B+, so I suspect that PNG is just really slow to encode.

[Eichhorn18]

...you may find it faster to do it on the ARM cores than asking the GPU to do it (250MHz processor vs 1.4GHz processor). That also allows the GPU to move on to capturing the next still for you.

Can you please explain how this could be achieved? I'm not aware of a way to direct a process from python script to the CPU vs. GPU. How can this be done?

Thanks

[6by9]

...you may find it faster to do it on the ARM cores than asking the GPU to do it (250MHz processor vs 1.4GHz processor). That also allows the GPU to move on to capturing the next still for you.

Can you please explain how this could be achieved? I'm not aware of a way to direct a process from python script to the CPU vs. GPU. How can this be done?

Get the RGB frame back from the GPU.
Find the relevant libPNG wrapper for Python (https://pythonhosted.org/pypng/png.html ?)
Pass the RGB frame into PNG library.

It looks like https://pythonhosted.org/pypng/ex.html# ... ng-writing may give you one solution using pypng.