informed
Posts: 121
Joined: Mon Oct 26, 2015 6:24 pm

improve quality of jpeg captured from mjpeg stream

Wed Nov 28, 2018 10:36 am

I am on the RPI and I am using raspivid to stream a video and at the same time extract jpegs from the stream.

This I can do. But the quality of the jpeg is poor/dull.

Any ideas how to improve the quality? I have spent some days Googling and trying things out...

This are my parameters:

--codec MJPEG -vf -hf -b 200000000 -w 810 -h 648 -fps 10 --nopreview -t 0 -o -

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6055
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: improve quality of jpeg captured from mjpeg stream

Wed Nov 28, 2018 12:19 pm

200Mbit/s?!!? So you're wanting 1.25MB JPEGs. Running it myself I see the message
"Bitrate too high: Reducing to 25Mbit/s"
so that's what is actually be used. Approx 320kB per image.

Within the firmware there is a limit within the MJPEG codec of using JPEG quality 20. Enabling the logging I see that it is using quality 20 and producing 76kB images when the target is 312kB. I suspect that that limit was added to stop bitrate control going mad and just cranking up the quality and significantly overshooting (eg if the lights are off in a room so quality can be very high, then the lights come on and it suddenly encodes a huge frame).
That limit could be increased as I see it being hit even with 720P 10fps @ 25Mbit/s. I'll have a think on that one.

To be honest you should really change the app to use image_encode instead of video_encode, and then you can set the quality to a value of your choosing. That's what you're really after as you appear to want individual JPEGs.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

informed
Posts: 121
Joined: Mon Oct 26, 2015 6:24 pm

Re: improve quality of jpeg captured from mjpeg stream

Wed Nov 28, 2018 12:24 pm

HI

Thanks for the info.

I need to get the images from the stream as real time as possible as I am trying (and getting 10fps).

I knew I should have enabled verbose :)

Is there anyway to override the q of '20' to something higher?

Thanks Again (I am still learning)

informed
Posts: 121
Joined: Mon Oct 26, 2015 6:24 pm

Re: improve quality of jpeg captured from mjpeg stream

Wed Nov 28, 2018 12:26 pm

Forgot the reason for high bitrate is that I heard there was no performance loss setting that as it will only use what it needs to use?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6055
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: improve quality of jpeg captured from mjpeg stream

Wed Nov 28, 2018 1:54 pm

informed wrote:
Wed Nov 28, 2018 12:24 pm
I need to get the images from the stream as real time as possible as I am trying (and getting 10fps).
video_encode vs image_encode should make nearly no difference there. The MJPEG codec uses the same JPEG hardware block as JPEG. You will want to disable EXIF and thumbnails though.
Switching between camera->output[1] (video) and camera->output[2] (stills) will make a big difference.
informed wrote: I knew I should have enabled verbose :)
It wasn't needed - the message was printed anyway.
informed wrote: Is there anyway to override the q of '20' to something higher?
Not for you as it is in the firmware. I will check what happens if I increase it up to 60.
informed wrote:
Wed Nov 28, 2018 12:26 pm
Forgot the reason for high bitrate is that I heard there was no performance loss setting that as it will only use what it needs to use?
Not really.
You are setting the target bitrate. The codec has control over the JPEG Q-factor to try and achieve that. If it is allowed to then it will turn up the Q-factor to generate that bitrate, increasing the high frequency content as it goes up. Should there be insufficient high frequency content then the file size won't increase, but that is really only the case with a picture of a black cat on a dark night (and even then it'll just encode the noise more accurately).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

informed
Posts: 121
Joined: Mon Oct 26, 2015 6:24 pm

Re: improve quality of jpeg captured from mjpeg stream

Wed Nov 28, 2018 2:02 pm

Hi

Thanks again for your reply...

"video_encode vs image_encode should make nearly no difference there. The MJPEG codec uses the same JPEG hardware block as JPEG. You will want to disable EXIF and thumbnails though.
Switching between camera->output[1] (video) and camera->output[2] (stills) will make a big difference."


Forgive my ignorance here, Are you suggesting I use raspisitll? Becuase I had mucked around with it and the max I can get is 5fps?

Would you have an example of the commandline for me please?

"It wasn't needed - the message was printed anyway."

I was redirecting the standout in my code so was not displayed

Not for you as it is in the firmware. I will check what happens if I increase it up to 60

Thanks. Though I did include the -q flag in the commandline using raspivid and the stream did not start :(

Not really.
You are setting the target bitrate. The codec has control over the JPEG Q-factor to try and achieve that. If it is allowed to then it will turn up the Q-factor to generate that bitrate, increasing the high frequency content as it goes up. Should there be insufficient high frequency content then the file size won't increase, but that is really only the case with a picture of a black cat on a dark night (and even then it'll just encode the noise more accurately).


Thanks for the lesson. I was only going by what I read. Though your explanation makes sense :)

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6055
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: improve quality of jpeg captured from mjpeg stream

Wed Nov 28, 2018 2:07 pm

informed wrote:
Wed Nov 28, 2018 2:02 pm
"video_encode vs image_encode should make nearly no difference there. The MJPEG codec uses the same JPEG hardware block as JPEG. You will want to disable EXIF and thumbnails though.
Switching between camera->output[1] (video) and camera->output[2] (stills) will make a big difference."


Forgive my ignorance here, Are you suggesting I use raspisitll? Becuase I had mucked around with it and the max I can get is 5fps?

Would you have an example of the commandline for me please?
No, I'm suggesting you modify the app. There isn't a standard app that does this.
informed wrote:"It wasn't needed - the message was printed anyway."

I was redirecting the standout in my code so was not displayed
It's printed to stderr
https://github.com/raspberrypi/userland ... id.c#L2096
informed wrote:Not for you as it is in the firmware. I will check what happens if I increase it up to 60

Thanks. Though I did include the -q flag in the commandline using raspivid and the stream did not start :(
It won't as -q is not a valid parameter to raspivid.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

informed
Posts: 121
Joined: Mon Oct 26, 2015 6:24 pm

Re: improve quality of jpeg captured from mjpeg stream

Wed Nov 28, 2018 2:10 pm

>No, I'm suggesting you modify the app. There isn't a standard app that does this.

i am sorry I do not know what you mean. I am using C#

>It's printed to stderr

Yes I know (thanks for the link) I just had not coded it you see


>It won't as -q is not a valid parameter to raspivid.

Yes, realised that after doing it and looking at the docs. Are you the developer of Raspivid?

Thnaks

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6055
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: improve quality of jpeg captured from mjpeg stream

Wed Nov 28, 2018 2:57 pm

informed wrote:
Wed Nov 28, 2018 2:10 pm
>No, I'm suggesting you modify the app. There isn't a standard app that does this.

i am sorry I do not know what you mean. I am using C#
Modify raspivid.
informed wrote:Yes, realised that after doing it and looking at the docs. Are you the developer of Raspivid?
I'm one of the developers at Pi Towers who does some stuff with raspivid (and the firmware, and the kernel, and almost anything else).
No, I'm not going to make that sort of change for you, and it won't be merged back into the standard one.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

informed
Posts: 121
Joined: Mon Oct 26, 2015 6:24 pm

Re: improve quality of jpeg captured from mjpeg stream

Wed Nov 28, 2018 2:59 pm

Ok - fair enough.

Thanks

informed
Posts: 121
Joined: Mon Oct 26, 2015 6:24 pm

Re: improve quality of jpeg captured from mjpeg stream

Wed Nov 28, 2018 4:18 pm

Sorry to trouble you again. I need to ask am O allowed to use the raspivid and raspistill in a commercial application?

Not intending to resell this app. I am just selling (or attempting to sell) a service that categories motion frames,

thanks

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 21143
Joined: Sat Jul 30, 2011 7:41 pm

Re: improve quality of jpeg captured from mjpeg stream

Wed Nov 28, 2018 4:29 pm

informed wrote:
Wed Nov 28, 2018 4:18 pm
Sorry to trouble you again. I need to ask am O allowed to use the raspivid and raspistill in a commercial application?

Not intending to resell this app. I am just selling (or attempting to sell) a service that categories motion frames,

thanks
Yes.

Even if you did sell it, as long as you provide the source code for any changes (since they have a GPL licence), you can still do so.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

informed
Posts: 121
Joined: Mon Oct 26, 2015 6:24 pm

Re: improve quality of jpeg captured from mjpeg stream

Wed Nov 28, 2018 4:33 pm

OK - thanks. Just making sure I am doing the right things.

Out of interest (and I do a appreciate the personal time you have shown me so far) did you actually see if raising the quality in the firmware itself made a difference at all?

I am looking at gstream, ffmpeg, vlc and MMAL itself to see what is possible so just asking people these type of questions as well as my own testing which is where I had got to with the raspivid.

-

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6055
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: improve quality of jpeg captured from mjpeg stream

Wed Nov 28, 2018 4:45 pm

jamesh wrote:
Wed Nov 28, 2018 4:29 pm
informed wrote:
Wed Nov 28, 2018 4:18 pm
Sorry to trouble you again. I need to ask am O allowed to use the raspivid and raspistill in a commercial application?

Not intending to resell this app. I am just selling (or attempting to sell) a service that categories motion frames,

thanks
Yes.

Even if you did sell it, as long as you provide the source code for any changes (since they have a GPL licence), you can still do so.
GPL?
https://github.com/raspberrypi/userland ... RaspiVid.c

Code: Select all

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
    * Redistributions of source code must retain the above copyright
      notice, this list of conditions and the following disclaimer.
    * Redistributions in binary form must reproduce the above copyright
      notice, this list of conditions and the following disclaimer in the
      documentation and/or other materials provided with the distribution.
    * Neither the name of the copyright holder nor the
      names of its contributors may be used to endorse or promote products
      derived from this software without specific prior written permission.
That's not GPL. No requirement to publish the modified source at all, but should you do so then you can't remove the copyright terms.

You're pretty much free to do what the heck you like with it.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.


6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6055
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: improve quality of jpeg captured from mjpeg stream

Wed Nov 28, 2018 5:22 pm

informed wrote:
Wed Nov 28, 2018 4:33 pm
Out of interest (and I do a appreciate the personal time you have shown me so far) did you actually see if raising the quality in the firmware itself made a difference at all?

I am looking at gstream, ffmpeg, vlc and MMAL itself to see what is possible so just asking people these type of questions as well as my own testing which is where I had got to with the raspivid.
I hadn't tried it. It seems to work, although I've also just seen it lock up on quit. I thought we'd fixed that one.

GStreamer, ffmpeg, and VLC will all use either OpenMax or MMAL if they want to use the hardware acceleration, which means they use the same firmware code.

Having said I wouldn't make the changes to raspivid for you, I tried it and it worked.
"sudo apt install cmake"
"git clone https://github.com/raspberrypi/userland"
Apply the patch

Code: Select all

diff --git a/host_applications/linux/apps/raspicam/RaspiVid.c b/host_applications/linux/apps/raspicam/RaspiVid.c
index 21fdc2e..b968c9e 100755
--- a/host_applications/linux/apps/raspicam/RaspiVid.c
+++ b/host_applications/linux/apps/raspicam/RaspiVid.c
@@ -1781,7 +1782,7 @@ static MMAL_STATUS_T create_camera_component(RASPIVID_STATE *state)
         mmal_port_parameter_set(video_port, &fps_range.hdr);
    }
 
-   format->encoding = MMAL_ENCODING_OPAQUE;
+   format->encoding = MMAL_ENCODING_I420;
    format->es->video.width = VCOS_ALIGN_UP(state->width, 32);
    format->es->video.height = VCOS_ALIGN_UP(state->height, 16);
    format->es->video.crop.x = 0;
@@ -2046,7 +2047,7 @@ static MMAL_STATUS_T create_encoder_component(RASPIVID_STATE *state)
    MMAL_STATUS_T status;
    MMAL_POOL_T *pool;
 
-   status = mmal_component_create(MMAL_COMPONENT_DEFAULT_VIDEO_ENCODER, &encoder);
+   status = mmal_component_create(MMAL_COMPONENT_DEFAULT_IMAGE_ENCODER, &encoder);
 
    if (status != MMAL_SUCCESS)
    {
@@ -2064,11 +2065,16 @@ static MMAL_STATUS_T create_encoder_component(RASPIVID_STATE *state)
    encoder_input = encoder->input[0];
    encoder_output = encoder->output[0];
 
+   status = mmal_port_parameter_set_boolean(encoder_output, MMAL_PARAMETER_EXIF_DISABLE, MMAL_TRUE);
+   status = mmal_port_parameter_set_uint32(encoder_output, MMAL_PARAMETER_JPEG_Q_FACTOR, 40);
+   MMAL_PARAMETER_THUMBNAIL_CONFIG_T param_thumb = {{MMAL_PARAMETER_THUMBNAIL_CONFIGURATION, sizeof(MMAL_PARAMETER_THUMBNAIL_CONFIG_T)}, 0, 0, 0, 0};
+   status = mmal_port_parameter_set(encoder->control, &param_thumb.hdr);
+
    // We want same format on input and output
    mmal_format_copy(encoder_output->format, encoder_input->format);
 
    // Only supporting H264 at the moment
-   encoder_output->format->encoding = state->encoding;
+   encoder_output->format->encoding = MMAL_ENCODING_JPEG; //state->encoding;
 
    if(state->encoding == MMAL_ENCODING_H264)
    {
"./buildme" to build and install everything. Retry your raspivid line (don't expect it to work should you omit the "--codec MJPEG" option).
It's hardcoded to set MMAL_PARAMETER_JPEG_Q_FACTOR to 40 - alter as you wish.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

informed
Posts: 121
Joined: Mon Oct 26, 2015 6:24 pm

Re: improve quality of jpeg captured from mjpeg stream

Wed Nov 28, 2018 5:34 pm

Thanks!

I was going to offer you £20 to do it for me as well but did not know how to approach the subject.

I can still do that if you give me your PayPal email?

Thanks Again :)

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6055
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: improve quality of jpeg captured from mjpeg stream

Wed Nov 28, 2018 5:51 pm

informed wrote:
Wed Nov 28, 2018 5:34 pm
I was going to offer you £20 to do it for me as well but did not know how to approach the subject.
Note my sig - if it's not done as part of the day job then I'm not really interested in doing it.

I think I've found the bug in the MJPEG encoder that is causing it to lock up. I'm sort of surprised it worked as well seeing as it is quite such a shocker!
I'll see what I can do about fixing it up.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

informed
Posts: 121
Joined: Mon Oct 26, 2015 6:24 pm

Re: improve quality of jpeg captured from mjpeg stream

Wed Nov 28, 2018 5:54 pm

I know, I did read that that was why I did not know how to approach the subject. I did not want to come across as insulting to you and why I was looking at other frameworks,

This is all a learning curve to me - which I love - and I have already learnt a lot from you during my time on this forum,

Impossible to thank you enough...

informed
Posts: 121
Joined: Mon Oct 26, 2015 6:24 pm

Re: improve quality of jpeg captured from mjpeg stream

Wed Nov 28, 2018 5:58 pm

please confirm the arguments (as video is not streaming):

I have :

--codec MJPEG -q 75 -vf -hf -b 200000000 -w 640 -h 480 -fps 10 -t 0 -o -"

Told you I am a beginner

informed
Posts: 121
Joined: Mon Oct 26, 2015 6:24 pm

Re: improve quality of jpeg captured from mjpeg stream

Wed Nov 28, 2018 5:59 pm

yh - please ignore my bitrate - u had changed it

informed
Posts: 121
Joined: Mon Oct 26, 2015 6:24 pm

Re: improve quality of jpeg captured from mjpeg stream

Wed Nov 28, 2018 7:03 pm

Hi,

Just realised I had to put the patch in myself lol.

Wish I learnt c and c++ looks much more fun than c#.

I put the changes in but was nt sure abit 2 of the lines.

I run the build and I got this error:

recipe for RaspiVid.c.o failed. Error 1
recipe for target all failed Error 2

So sorry to be a pain... I wil keep on trying...

This is my amended RaspiVid.c script:

informed
Posts: 121
Joined: Mon Oct 26, 2015 6:24 pm

Re: improve quality of jpeg captured from mjpeg stream

Wed Nov 28, 2018 7:12 pm

ignore last message.

I made the mistake of including these lines:

@@ -2064,11 +2065,16 @@ static MMAL_STATUS_T create_encoder_component(RASPIVID_STATE *state) etc...

What are those lines anyway? I did google it.

I think i would like to learn c.. Looks like proper programming to me, What IDE do you use?

thanks

informed
Posts: 121
Joined: Mon Oct 26, 2015 6:24 pm

Re: improve quality of jpeg captured from mjpeg stream

Wed Nov 28, 2018 7:39 pm

So sorry to keep badgering you.

everything runs OK if I output to screen but if I try and redirect it to standout it fails. I stripped the args o basic:

-w 810 -h 648 -fps 10 -codec MJPEG -t 0 -o -

No bytes are received in my pipe. Is there anything that can interfere with that? I cannot see how though but thought I would ask. Will obviously keep on trying :)

informed
Posts: 121
Joined: Mon Oct 26, 2015 6:24 pm

Re: improve quality of jpeg captured from mjpeg stream

Wed Nov 28, 2018 7:56 pm

Hi,

I bet you wished you never replied to me in the 1st place now? I have nver touched C and I think I must have implemented the patch wrong.

I have hooked up the standard error message (I know i was lazy not to have done so in the 1st place_.

The error I get from that is:

mmal: Unable to set immutable input flag

Return to “Camera board”