richie47
Posts: 2
Joined: Thu Oct 03, 2019 4:18 pm

Raspi 4b reserve core for C++

Wed Nov 27, 2019 5:47 pm

Does anyone have an example of reserving a core(s) for software on the raspberry pi 4b?

thanks

swampdog
Posts: 301
Joined: Fri Dec 04, 2015 11:22 am

Re: Raspi 4b reserve core for C++

Thu Nov 28, 2019 1:56 am

Can't remember how to use it but here you go..

Code: Select all

admin@pi05:~ $ apropos affinity
pthread_attr_getaffinity_np (3) - set/get CPU affinity attribute in thread at...
pthread_attr_setaffinity_np (3) - set/get CPU affinity attribute in thread at...
pthread_getaffinity_np (3) - set/get CPU affinity of a thread
pthread_setaffinity_np (3) - set/get CPU affinity of a thread
sched_getaffinity (2) - set and get a thread's CPU affinity mask
sched_setaffinity (2) - set and get a thread's CPU affinity mask
taskset (1)          - retrieve or set a process's CPU affinity

Heater
Posts: 14297
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspi 4b reserve core for C++

Thu Nov 28, 2019 8:52 am

You can stop Linux from scheduling anything on a core or cores by using the "isolcpus" option on the kernel boot command in /boot/cmdline.txt
https://yosh.ke.mu/raspberry_pi_isolati ... nux_kernel

Then you can run your program all by itself on that free core by starting it with the "taskset" command.

Or use the pthreads calls indicated above to get threads to run on that free core.
Memory in C++ is a leaky abstraction .

dsyleixa123
Posts: 578
Joined: Mon Jun 11, 2018 11:22 am

Re: Raspi 4b reserve core for C++

Thu Nov 28, 2019 8:58 am

is Pi4 in this respect different from earlier ones? I once got this proposal for my Pi2 here in this forum:

Code: Select all

// run program on reserved cpus 2+3
//
// add in /boot/cmdline.txt :
// isolcpus= 2,3
// https://www.raspberrypi.org/forums/viewtopic.php?f=33&t=247493&p=1512885#p1512885
// You can check by looking at /proc/<PID of your program>/status 
// and looking at the Cpus_allowed_list line.


#define _GNU_SOURCE

#include <stdio.h>
#include <stdlib.h>
#include <sched.h>

int main()
{
  cpu_set_t cpus;

  CPU_ZERO(&cpus);

  CPU_SET(2, &cpus);
  CPU_SET(3, &cpus);

  if (sched_setaffinity(0, sizeof(cpus), &cpus)) {
    perror("sched_setaffinity");
    exit(1);
  }
  
  /* Do stuff */
}

Heater
Posts: 14297
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspi 4b reserve core for C++

Thu Nov 28, 2019 11:03 am

Looks good to me. No reason it should be different on the Pi 4.
Memory in C++ is a leaky abstraction .

User avatar
MikeDB
Posts: 166
Joined: Sun Oct 12, 2014 8:27 am

Re: Raspi 4b reserve core for C++

Thu Nov 28, 2019 11:08 am

In main() after any initialisation put

pthread_t thread0, thread1, thread2, thread3;
const char *message0 = "Starting 0";
const char *message1 = "Starting 1";
const char *message2 = "Starting 2";
const char *message3 = "Starting 3";
int iret0, iret1, iret2, iret3;

// Create independent threads each of which will execute function

iret0 = pthread_create(&thread0, NULL, function0, (void*) message0);
if (iret0)
{
fprintf(stderr,"Thread 0 return : %d\n",iret0);
exit(EXIT_FAILURE);
}

Same for iret1, 2 and 3
Then use :

// All threads should never finish
pthread_join(thread0, NULL);
pthread_join(thread1, NULL);
pthread_join(thread2, NULL);
pthread_join(thread3, NULL);



Then for each thread you want

void *function0(void *ptr)
{
char *message = (char*) ptr;
printf("%s \n", message);
cpu_set_t cpuset;
CPU_ZERO(&cpuset);
CPU_SET(1,&cpuset);
sched_setaffinity(0, sizeof(cpu_set_t), &cpuset);
while (1)
DoStuff();
}


Finally you use isolcpu = 1,2,3 or whatever you need to isolate the time critical threads. Put all your non-time critcal stuff in one of the threads you don't isolate and allow Linux to also use that CPU or of course you will have real problems.

This will give you over 99.9% of a CPU time, but there still seems to be an occasional interrupt which just returns. Still tracing this but I think it is something to do with memory refresh or management - maybe a Pi employee can comment.
Will the forthcoming MIDI-2 spec at last allow us to set the volume to 11 !!

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24699
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raspi 4b reserve core for C++

Thu Nov 28, 2019 12:56 pm

MikeDB wrote:
Thu Nov 28, 2019 11:08 am
In main() after any initialisation put

pthread_t thread0, thread1, thread2, thread3;
const char *message0 = "Starting 0";
const char *message1 = "Starting 1";
const char *message2 = "Starting 2";
const char *message3 = "Starting 3";
int iret0, iret1, iret2, iret3;

// Create independent threads each of which will execute function

iret0 = pthread_create(&thread0, NULL, function0, (void*) message0);
if (iret0)
{
fprintf(stderr,"Thread 0 return : %d\n",iret0);
exit(EXIT_FAILURE);
}

Same for iret1, 2 and 3
Then use :

// All threads should never finish
pthread_join(thread0, NULL);
pthread_join(thread1, NULL);
pthread_join(thread2, NULL);
pthread_join(thread3, NULL);



Then for each thread you want

void *function0(void *ptr)
{
char *message = (char*) ptr;
printf("%s \n", message);
cpu_set_t cpuset;
CPU_ZERO(&cpuset);
CPU_SET(1,&cpuset);
sched_setaffinity(0, sizeof(cpu_set_t), &cpuset);
while (1)
DoStuff();
}


Finally you use isolcpu = 1,2,3 or whatever you need to isolate the time critical threads. Put all your non-time critcal stuff in one of the threads you don't isolate and allow Linux to also use that CPU or of course you will have real problems.

This will give you over 99.9% of a CPU time, but there still seems to be an occasional interrupt which just returns. Still tracing this but I think it is something to do with memory refresh or management - maybe a Pi employee can comment.
Not sure what the interrupt might be, perhaps DMA or similar. Anyway, if your system is so time critical that a stubbed interrupt on a 1500Mhz core causes problems, then you need a faster core. Do you encounter any issues with the ARM core speed changing under load and temperature?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

dsyleixa123
Posts: 578
Joined: Mon Jun 11, 2018 11:22 am

Re: Raspi 4b reserve core for C++

Thu Nov 28, 2019 1:04 pm

MikeDB wrote:
Thu Nov 28, 2019 11:08 am
In main() after any initialisation put

pthread_t thread0, thread1, thread2, thread3;
const char *message0 = "Starting 0";
const char *message1 = "Starting 1";
const char *message2 = "Starting 2";
const char *message3 = "Starting 3";
int iret0, iret1, iret2, iret3;

// Create independent threads each of which will execute function

iret0 = pthread_create(&thread0, NULL, function0, (void*) message0);
if (iret0)
{
fprintf(stderr,"Thread 0 return : %d\n",iret0);
exit(EXIT_FAILURE);
}

Same for iret1, 2 and 3
Then use :

// All threads should never finish
pthread_join(thread0, NULL);
pthread_join(thread1, NULL);
pthread_join(thread2, NULL);
pthread_join(thread3, NULL);



Then for each thread you want

void *function0(void *ptr)
{
char *message = (char*) ptr;
printf("%s \n", message);
cpu_set_t cpuset;
CPU_ZERO(&cpuset);
CPU_SET(1,&cpuset);
sched_setaffinity(0, sizeof(cpu_set_t), &cpuset);
while (1)
DoStuff();
}


Finally you use isolcpu = 1,2,3 or whatever you need to isolate the time critical threads. Put all your non-time critcal stuff in one of the threads you don't isolate and allow Linux to also use that CPU or of course you will have real problems.

This will give you over 99.9% of a CPU time, but there still seems to be an occasional interrupt which just returns. Still tracing this but I think it is something to do with memory refresh or management - maybe a Pi employee can comment.
@MikeDB:

this is really interesting stuff, but for me still a little confusing, tbh...
would you mind to provide a complete source code example (perhaps including my and your snippets) for
thread0: time critical
thread1: time critical

thread2: NOT time critical
thread3: NOT time critical

8-)

Heater
Posts: 14297
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspi 4b reserve core for C++

Thu Nov 28, 2019 6:36 pm

A better I idea would be to read the pthreads documentation and tutorials.

There is a lot of stuff about pthreads around the net. For example:
https://computing.llnl.gov/tutorials/pthreads/

Of course as you are using C++ you should probably be using std::thread instead:
https://en.cppreference.com/w/cpp/thread/thread
Memory in C++ is a leaky abstraction .

dsyleixa123
Posts: 578
Joined: Mon Jun 11, 2018 11:22 am

Re: Raspi 4b reserve core for C++

Thu Nov 28, 2019 8:16 pm

tbh, as a hobby programmer coming from Arduino, I am not a friend of man pages: intead, I learn from example codes, by doing and tinkering... 8-)

Heater
Posts: 14297
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspi 4b reserve core for C++

Thu Nov 28, 2019 10:45 pm

Lucky I did not link you to any man mages then.

That first link is actually a very nice tutorial with loads of ready made examples. Including all the bits of code they are talking about highlighted in Red.

But a google search for "pthreads for dummies" will find you thousands of even simpler examples. For example: https://vcansimplify.wordpress.com/2013 ... implified/

No matter what hobby you pursue I recommend learning to read first. It helped me a lot when I was trying to get my head around electronics and digital logic as a kid.
Memory in C++ is a leaky abstraction .

LdB
Posts: 1374
Joined: Wed Dec 07, 2016 2:29 pm

Re: Raspi 4b reserve core for C++

Fri Nov 29, 2019 12:11 am

jamesh wrote:
Thu Nov 28, 2019 12:56 pm
Not sure what the interrupt might be, perhaps DMA or similar. Anyway, if your system is so time critical that a stubbed interrupt on a 1500Mhz core causes problems, then you need a faster core. Do you encounter any issues with the ARM core speed changing under load and temperature
You need to first set the IRQ Affinity, so that the isolated CPU doesn’t receive any interrupts. Even if you do all that any RP_PREEMPT task from linux is allowed to use the core unless you are using the RT kernel.

Heater
Posts: 14297
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspi 4b reserve core for C++

Fri Nov 29, 2019 4:17 am

LdB,
You need to first set the IRQ Affinity, so that the isolated CPU doesn’t receive any interrupts.
Do you have any links to docs/advie about that?
...any RP_PREEMPT task from linux is allowed to use the core unless you are using the RT kernel.
What is "RP_PREEMPT". I have never heard of it and google does not know.

In another thread all interrupts to isolated cores were killed off by sneakily writing to some interrupt control register or vector table. Sadly I forget the details and can'f find the post just now.

Even without that trick I was only seeing a 5us interruption to my wiggling of a GPIO pin at 50MHz every couple of milliseconds.

I'd love to know what that interrupt is an how to kill it off properly. Then one would have use of an isolated core as if you were running bare metal.
Memory in C++ is a leaky abstraction .

dsyleixa123
Posts: 578
Joined: Mon Jun 11, 2018 11:22 am

Re: Raspi 4b reserve core for C++

Fri Nov 29, 2019 8:30 am

Basically I'm fine with pthread (even for C++), but as stated here
@MikeDB:

this is really interesting stuff, but for me still a little confusing, tbh...
would you mind to provide a complete source code example (perhaps including my and your snippets) for
thread0: time critical
thread1: time critical
thread2: NOT time critical
thread3: NOT time critical
for a first access it would be very nice to have this framework
https://www.raspberrypi.org/forums/view ... 4#p1572837
merged with this one
https://www.raspberrypi.org/forums/view ... 4#p1572916

having that, one could try and test and tinker araund with some different proprietary changes and settings...

Heater
Posts: 14297
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspi 4b reserve core for C++

Fri Nov 29, 2019 8:43 am

dsyleixa123,

I'm confused.

Are you fine with it or are you confused?

Either way we can't be spoon feeding people with large pieces of example code custom made to cater for every little query.

Besides what you ask for exists in spades already, in the examples in the links I gave and many others all over the net.

It's time to roll your sleeves up, dive in, and do some good old fashioned homework.
Memory in C++ is a leaky abstraction .

dsyleixa123
Posts: 578
Joined: Mon Jun 11, 2018 11:22 am

Re: Raspi 4b reserve core for C++

Fri Nov 29, 2019 8:49 am

I am fine with pthread so far, but confused about how to merge MikeDB's code into mine, in order not only to reserve cores for the program from Linux, but also to assign even single arbitrary threads to related cores. What is so hard to understand about it?

Heater
Posts: 14297
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspi 4b reserve core for C++

Fri Nov 29, 2019 9:26 am

dsyleixa123,
...confused about how to merge MikeDB's code into mine, in order not only to reserve cores for the program from Linux,...
MikeDB's code does not do anything about "reserve cores for the program".

One can prevent the Linux kernel from scheduling tasks on a core or cores using the "isolcpus" parameter on the Linux boot command.

As I explained in this very thread: https://www.raspberrypi.org/forums/view ... 3#p1572835

Then one can run a thread or threads on those reserved cores by using the "sched_setaffinity" function.

As you say you are fine with phreads I will assume you have code that gets some threads started. Then all you need to do is make calls to the sched_setaffinity() at the begining of your thread functions.

As MikeDB shows (With annotations by me)

Code: Select all

    // Then for each thread you want

    void *function0(void *ptr)                              // A function run as as a pthread.
    {
        char *message = (char*) ptr;                        // Start up message for thread.                
        printf("%s \n", message);

        cpu_set_t cpuset;                                   // Code to cause this thread to be run on
        CPU_ZERO(&cpuset);                                  // a particular core.
        CPU_SET(1,&cpuset);                                 // Add this to your thread functions with suitable parameter adjustments.
        sched_setaffinity(0, sizeof(cpu_set_t), &cpuset);   //
  
        while (1)                                           // The actual work of the thread.
            DoStuff();
    }
Memory in C++ is a leaky abstraction .

dsyleixa123
Posts: 578
Joined: Mon Jun 11, 2018 11:22 am

Re: Raspi 4b reserve core for C++

Fri Nov 29, 2019 9:35 am

my question is just about this statement:
Put all your non-time critcal stuff in one of the threads you don't isolate
thats why I asked
for a first access it would be very nice to have this framework
https://www.raspberrypi.org/forums/view ... 4#p1572837
merged with this one
https://www.raspberrypi.org/forums/view ... 4#p1572916
thread0: time critical
thread1: time critical
thread2: NOT time critical
thread3: NOT time critical

Heater
Posts: 14297
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspi 4b reserve core for C++

Fri Nov 29, 2019 11:12 am

Now I have no idea what you are asking exactly. Could restate the question.

Surely "Put all your non-time critcal stuff in one of the threads you don't isolate" is clear enough.

You have a core or cores you have isolated from the kernel scheduler with isolcpus, as shown above.

You cause your time critical threads to be run on those isolated cores by using "sched_setaffinity", as shown above.

For good measure cause your non-time critical threads to be run on the non-isolated cores by using "sched_setaffinity", as shown above.

What am I missing here?
Memory in C++ is a leaky abstraction .

dsyleixa123
Posts: 578
Joined: Mon Jun 11, 2018 11:22 am

Re: Raspi 4b reserve core for C++

Fri Nov 29, 2019 11:57 am

What am I missing here?
perhaps the actuall entire source code...? ;)

Heater
Posts: 14297
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspi 4b reserve core for C++

Fri Nov 29, 2019 12:13 pm

I'm sorry, but as much as I like to help I cannot be writing and testing large chunks of code just to answer a simple forum question. I have difficulty enough finding time to write code for my boss, our clients and myself!

You have said you are fine with pthreads so I assume you have code with some threads running. All that is required then is the four lines of code I pointed out above to be added to the beginning of each thread function.

And the setting of that isolcpus kernel boot parameter as described above.

Be curious, have fun, try it for yourself!

Oh, and read the man pages :)
Memory in C++ is a leaky abstraction .

dsyleixa123
Posts: 578
Joined: Mon Jun 11, 2018 11:22 am

Re: Raspi 4b reserve core for C++

Fri Nov 29, 2019 12:36 pm

ok, my 1st approach would be this, without completely detailed cpu settings

Code: Select all

// run program on reserved cpus 2+3
//
// add in /boot/cmdline.txt :
// isolcpus= 2,3
// https://www.raspberrypi.org/forums/viewtopic.php?f=33&t=247493&p=1512885#p1512885
// You can check by looking at /proc/<PID of your program>/status 
// and looking at the Cpus_allowed_list line.


#define _GNU_SOURCE

#include <stdio.h>
#include <stdlib.h>

#include <sched.h>
#include <pthread.h>

cpu_set_t cpus;



void *function0(void *ptr)                              // A function run as as a pthread.
{
        char *message = (char*) ptr;                        // Start up message for thread.                
        printf("%s \n", message);

        cpu_set_t cpuset;                                   // Code to cause this thread to be run on
        CPU_ZERO(&cpuset);                                  // a particular core.
        CPU_SET(1,&cpuset);                                 // Add this to your thread functions with suitable parameter adjustments.
        sched_setaffinity(0, sizeof(cpu_set_t), &cpuset);   //
  
        while (1)                                           // The actual work of the thread.
        {
            DoStuff();
        }
}


void *function1(void *ptr)                              // A function run as as a pthread.
{
        char *message = (char*) ptr;                        // Start up message for thread.                
        printf("%s \n", message);

        cpu_set_t cpuset;                                   // Code to cause this thread to be run on
        CPU_ZERO(&cpuset);                                  // a particular core.
        CPU_SET(1,&cpuset);                                 // Add this to your thread functions with suitable parameter adjustments.
        sched_setaffinity(0, sizeof(cpu_set_t), &cpuset);   //
  
        while (1)                                           // The actual work of the thread.
        {
            DoStuff();
        }
}



void *function2(void *ptr)                              // A function run as as a pthread.
{
        char *message = (char*) ptr;                        // Start up message for thread.                
        printf("%s \n", message);

        cpu_set_t cpuset;                                   // Code to cause this thread to be run on
        CPU_ZERO(&cpuset);                                  // a particular core.
        CPU_SET(1,&cpuset);                                 // Add this to your thread functions with suitable parameter adjustments.
        sched_setaffinity(0, sizeof(cpu_set_t), &cpuset);   //
  
        while (1)                                           // The actual work of the thread.
        {
            DoStuff();
        }
}



void *function3(void *ptr)                              // A function run as as a pthread.
{
        char *message = (char*) ptr;                        // Start up message for thread.                
        printf("%s \n", message);

        cpu_set_t cpuset;                                   // Code to cause this thread to be run on
        CPU_ZERO(&cpuset);                                  // a particular core.
        CPU_SET(1,&cpuset);                                 // Add this to your thread functions with suitable parameter adjustments.
        sched_setaffinity(0, sizeof(cpu_set_t), &cpuset);   //
  
        while (1)                                           // The actual work of the thread.
        {
            DoStuff();
        }
}








int main()
{  

  CPU_ZERO(&cpus);

  CPU_SET(2, &cpus);
  CPU_SET(3, &cpus);

  if (sched_setaffinity(0, sizeof(cpus), &cpus)) {
    perror("sched_setaffinity");
    exit(1);
  }

  pthread_t thread0, thread1, thread2, thread3;

  const char *message0 = "Starting 0";
  const char *message1 = "Starting 1";
  const char *message2 = "Starting 2";
  const char *message3 = "Starting 3";
  
  int iret0, iret1, iret2, iret3;

  iret0 = pthread_create(&thread0, NULL, function0, (void*) message0);
  if (iret0)
  {
    fprintf(stderr,"Thread 0 return : %d\n",iret0);
    exit(EXIT_FAILURE);
  }
  
  iret1 = pthread_create(&thread1, NULL, function1, (void*) message1);
  if (iret1)
  {
    fprintf(stderr,"Thread 1 return : %d\n",iret1);
    exit(EXIT_FAILURE);
  }

  iret2 = pthread_create(&thread2, NULL, function2, (void*) message2);
  if (iret2)
  {
    fprintf(stderr,"Thread 2 return : %d\n",iret0);
    exit(EXIT_FAILURE);
  }

  iret3 = pthread_create(&thread3, NULL, function3, (void*) message3);
  if (iret3)
  {
    fprintf(stderr,"Thread 3 return : %d\n",iret0);
    exit(EXIT_FAILURE);
  }
  
    /* Do stuff */


  pthread_join(thread0, NULL);
  pthread_join(thread1, NULL);
  pthread_join(thread2, NULL);
  pthread_join(thread3, NULL);

  return 0;
  
}


1st,
is this entirely correct?

2nd,
what does this mean (in main() )?
CPU_ZERO(&cpus);
CPU_SET(2, &cpus);
CPU_SET(3, &cpus);

and 3rd,
what is this, in either thread, and how to adjust it correctly?

cpu_set_t cpuset; // Code to cause this thread to be run on
CPU_ZERO(&cpuset); // a particular core.
CPU_SET(1,&cpuset); // Add this to your thread functions with suitable parameter adjustments.
sched_setaffinity(0, sizeof(cpu_set_t), &cpuset); //

(as stated, the detailed settings are still confusing, too many letters :? )

Heater
Posts: 14297
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspi 4b reserve core for C++

Fri Nov 29, 2019 4:43 pm

Can you compile it?

Does it run?

Have you checked that it does what you want?

When any of those has the answer "no" come back with error messages, or actual vs expected results.

Then we can think about fixing it.
Memory in C++ is a leaky abstraction .

dsyleixa123
Posts: 578
Joined: Mon Jun 11, 2018 11:22 am

Re: Raspi 4b reserve core for C++

Fri Nov 29, 2019 5:55 pm

I am not the OP, I have just asked MikeDB how to merge his snippet with mine correctly.

Heater
Posts: 14297
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspi 4b reserve core for C++

Fri Nov 29, 2019 6:40 pm

You are not the OP.

You did however ask the question.

I did answer comprehensively.

Seems I am wasting my time. Sorry.
Memory in C++ is a leaky abstraction .

Return to “C/C++”