dsyleixa123
Posts: 405
Joined: Mon Jun 11, 2018 11:22 am

compile Linux C programs to use cpu cores exclusively

Thu Aug 01, 2019 7:42 am

IIRC, I once read that Linux C(++) programs can be compiled to use cpu cores exclusively without enabling kernel access (1 reserved core or perhaps even 2 of 4) - but I can't find the related post any more -
who can help please? 8)

swampdog
Posts: 239
Joined: Fri Dec 04, 2015 11:22 am

Re: compile Linux C programs to use cpu cores exclusively

Thu Aug 01, 2019 8:08 am

The magic word in this case is "affinity"..

Code: Select all

admin@pi05:~ $ apropos affinity
pthread_attr_getaffinity_np (3) - set/get CPU affinity attribute in thread at...
pthread_attr_setaffinity_np (3) - set/get CPU affinity attribute in thread at...
pthread_getaffinity_np (3) - set/get CPU affinity of a thread
pthread_setaffinity_np (3) - set/get CPU affinity of a thread
sched_getaffinity (2) - set and get a thread's CPU affinity mask
sched_setaffinity (2) - set and get a thread's CPU affinity mask
taskset (1)          - retrieve or set a process's CPU affinity
On overview..
https://www.glennklockwood.com/hpc-how ... nity.html
..noting that 'numactl' doesn't exist on a pi and that I've never done it myself (only on Aix which is different) so you'll be advised to experiment.

dsyleixa123
Posts: 405
Joined: Mon Jun 11, 2018 11:22 am

Re: compile Linux C programs to use cpu cores exclusively

Thu Aug 01, 2019 8:10 am

thank you, but as stated, (IIRC) there once was already posted a solution to the topic question (using compiler flags, CMIIW)

Heater
Posts: 13355
Joined: Tue Jul 17, 2012 3:02 pm

Re: compile Linux C programs to use cpu cores exclusively

Thu Aug 01, 2019 12:22 pm

Even if it is possible it sounds like a really bad idea.

Better to use the "taskset" command with root privs.
Memory in C++ is a leaky abstraction .

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23688
Joined: Sat Jul 30, 2011 7:41 pm

Re: compile Linux C programs to use cpu cores exclusively

Thu Aug 01, 2019 12:46 pm

dsyleixa123 wrote:
Thu Aug 01, 2019 8:10 am
thank you, but as stated, (IIRC) there once was already posted a solution to the topic question (using compiler flags, CMIIW)
Not sure its possible with compiler flags, certainly not something I have heard of.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

dsyleixa123
Posts: 405
Joined: Mon Jun 11, 2018 11:22 am

Re: compile Linux C programs to use cpu cores exclusively

Sat Aug 03, 2019 11:15 am

Heater wrote:
Thu Aug 01, 2019 12:22 pm
Even if it is possible it sounds like a really bad idea.
what causes you to this rediculous comment?
finally I do not want to set the affinity to 1 or 2 cores but exclude the kernel from 1 or 2 cores.

trejan
Posts: 583
Joined: Tue Jul 02, 2019 2:28 pm

Re: compile Linux C programs to use cpu cores exclusively

Sat Aug 03, 2019 12:45 pm

dsyleixa123 wrote:
Sat Aug 03, 2019 11:15 am
finally I do not want to set the affinity to 1 or 2 cores but exclude the kernel from 1 or 2 cores.
There is no option when compiling something that will tell the kernel to reserve cores for sole usage of that program. The only way to do what you want is to use the isolcpus option in /boot/cmdline.txt which tells the kernel not to schedule any processes on the specified core unless it is explicitly set using CPU affinity and then use the functions swampdog listed. Make sure to check the interrupt affinity settings as well.

dsyleixa123
Posts: 405
Joined: Mon Jun 11, 2018 11:22 am

Re: compile Linux C programs to use cpu cores exclusively

Sat Aug 03, 2019 1:11 pm

trejan wrote:
Sat Aug 03, 2019 12:45 pm
dsyleixa123 wrote:
Sat Aug 03, 2019 11:15 am
finally I do not want to set the affinity to 1 or 2 cores but exclude the kernel from 1 or 2 cores.
There is no option when compiling something that will tell the kernel to reserve cores for sole usage of that program. The only way to do what you want is to use the isolcpus option in /boot/cmdline.txt which tells the kernel not to schedule any processes on the specified core unless it is explicitly set using CPU affinity and then use the functions swampdog listed. Make sure to check the interrupt affinity settings as well.
what would be the correct way then to use cores 0+1 for my program?
in /boot/cmdline.txt :
isolcpus= 0, 1

and what else additionally, specifically?

trejan
Posts: 583
Joined: Tue Jul 02, 2019 2:28 pm

Re: compile Linux C programs to use cpu cores exclusively

Sat Aug 03, 2019 1:31 pm

dsyleixa123 wrote:
Sat Aug 03, 2019 1:11 pm
what would be the correct way then to use cores 0+1 for my program?
in /boot/cmdline.txt :
isolcpus= 0, 1
Yes but I'd suggest not using CPU0 because the kernel prefers using it for interrupt handling. It will put everything on CPU0 unless something balances it. Look at /proc/interrupts. You can move them using /proc/irq/<interrupt number>/smp_affinity_list or irqaffinity in /boot/cmdline.txt but it is easier to just use a different CPU.

You can check which are CPUs are isolated in the scheduler by looking at /sys/devices/system/cpu/isolated
Last edited by trejan on Sat Aug 03, 2019 2:09 pm, edited 2 times in total.

Heater
Posts: 13355
Joined: Tue Jul 17, 2012 3:02 pm

Re: compile Linux C programs to use cpu cores exclusively

Sat Aug 03, 2019 1:32 pm

Use the "taskset" command to run your program on the core of your choice.
Linux taskset Command Tutorial for Beginners : https://www.howtoforge.com/linux-taskset-command/
Better not to isolate core 0.
Memory in C++ is a leaky abstraction .

dsyleixa123
Posts: 405
Joined: Mon Jun 11, 2018 11:22 am

Re: compile Linux C programs to use cpu cores exclusively

Sat Aug 03, 2019 1:53 pm

thank you, so probaly better
in /boot/cmdline.txt :
isolcpus= 2,3

but it's far beyond me what they suggest to do about taskset, that's completely weird and odd :shock: :o
(I want to use a static setting, for all of my C programs I ever will compile)

anyway, I'm quite sure that I remember to once have read sth about compiler flags for this purpose :?

trejan
Posts: 583
Joined: Tue Jul 02, 2019 2:28 pm

Re: compile Linux C programs to use cpu cores exclusively

Sat Aug 03, 2019 2:13 pm

dsyleixa123 wrote:
Sat Aug 03, 2019 1:53 pm
anyway, I'm quite sure that I remember to once have read sth about compiler flags for this purpose :?
I don't know of any option that does what you're asking for and it would be a strange design decision to do it that way. Even if there was a compiler option that specified your CPU affinity mask, the kernel would still schedule other processes on that CPU.

dsyleixa123
Posts: 405
Joined: Mon Jun 11, 2018 11:22 am

Re: compile Linux C programs to use cpu cores exclusively

Sat Aug 03, 2019 2:47 pm

ok, if not other way possible, then the isolcpus=2,3 way.
so what exactly to set then additionally, specifically, statically?

trejan
Posts: 583
Joined: Tue Jul 02, 2019 2:28 pm

Re: compile Linux C programs to use cpu cores exclusively

Sat Aug 03, 2019 2:57 pm

dsyleixa123 wrote:
Sat Aug 03, 2019 2:47 pm
ok, if not other way possible, then the isolcpus=2,3 way.
so what exactly to set then additionally, specifically, statically?
Call sched_setaffinity in your program or run your program using taskset.

What are you doing that needs this anyway?

Heater
Posts: 13355
Joined: Tue Jul 17, 2012 3:02 pm

Re: compile Linux C programs to use cpu cores exclusively

Sat Aug 03, 2019 4:44 pm

dsyleixa123,

You can run your program using taskset on core 0 like so:

$ taskset 0x01 myProgram

That "0x01" is known as a mask. Each bit set in the binary representation of the mask represents a core that the taskset will try to run your program on.

So "0x01 is the hexadecimal representation of the bits 00000001. Notice that the bit zero is set to 1, being the first bit that means run on core 0.

If you want to run on core 2 use:

$ taskset 0x02 myProgram

Where "0x02" is 00000010 in binary, the second bit is set, so core 1.

For core 3 use:

$ taskset 0x04 myProgram

Where the "0x04" is 0000100 in binary, the third bit is set, so core 2.

Etc...

Unless you are using threads in your program forget about all that sched_setaffinity stuff.

Do learn about binary and hexadecimal numbers, and some boolean logic, before writing any more code. Then you won't find things so "weird and odd".
Last edited by Heater on Sat Aug 03, 2019 6:33 pm, edited 1 time in total.
Memory in C++ is a leaky abstraction .

User avatar
rpdom
Posts: 15214
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: compile Linux C programs to use cpu cores exclusively

Sat Aug 03, 2019 5:00 pm

Heater wrote:
Sat Aug 03, 2019 4:44 pm
So "0x01 is the hexadecimal representation of the bits 00000001. Notice that the bit zero is set to 1, being the first bit that means run on core 0.

If you want to run on core 2 use:

$ taskset 0x02 myProgram

Where "0x02" is 00000010 in binary, the second bit is set, so core 2.
If the cores start at 0 and that is 0x01, then 0x02 would be core 1, 0x04 would be core 2 and 0x08 would be core 3, surely?

trejan
Posts: 583
Joined: Tue Jul 02, 2019 2:28 pm

Re: compile Linux C programs to use cpu cores exclusively

Sat Aug 03, 2019 5:04 pm

Heater wrote:
Sat Aug 03, 2019 4:44 pm
Unless you are using threads in your program forget about all that sched_setaffinity stuff.
They appear to want it to do this automatically every time so sched_setaffinity somewhere near the start of their program would do it. taskset is calling sched_setaffinity anyway. The thread ones start with pthread_

jahboater
Posts: 4689
Joined: Wed Feb 04, 2015 6:38 pm

Re: compile Linux C programs to use cpu cores exclusively

Sat Aug 03, 2019 5:40 pm

rpdom wrote:
Sat Aug 03, 2019 5:00 pm
Heater wrote:
Sat Aug 03, 2019 4:44 pm
So "0x01 is the hexadecimal representation of the bits 00000001. Notice that the bit zero is set to 1, being the first bit that means run on core 0.

If you want to run on core 2 use:

$ taskset 0x02 myProgram

Where "0x02" is 00000010 in binary, the second bit is set, so core 2.
If the cores start at 0 and that is 0x01, then 0x02 would be core 1, 0x04 would be core 2 and 0x08 would be core 3, surely?
I just use:-

taskset -c 0 myprogram

Heater
Posts: 13355
Joined: Tue Jul 17, 2012 3:02 pm

Re: compile Linux C programs to use cpu cores exclusively

Sat Aug 03, 2019 6:34 pm

rpdom,

Well spotted. I fixed. Typical programmer, off by one errors all over the place :)
Memory in C++ is a leaky abstraction .

User avatar
rpdom
Posts: 15214
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: compile Linux C programs to use cpu cores exclusively

Sat Aug 03, 2019 7:14 pm

Heater wrote:
Sat Aug 03, 2019 6:34 pm
rpdom,

Well spotted. I fixed. Typical programmer, off by one errors all over the place :)
No worries. I've spent years of switching between languages that start arrays at 0 and some that start at 1.

dsyleixa123
Posts: 405
Joined: Mon Jun 11, 2018 11:22 am

Re: compile Linux C programs to use cpu cores exclusively

Sat Aug 03, 2019 8:14 pm

I actually do not want to start my program manually using taskset,
I want to compile them in a way so that all my programs always start automatically using cores 2+3, after having set
isolcpus=2,3
for all of my programs (always running on both cores 2+3 simultaneously, the kernel always on 0+1)

trejan
Posts: 583
Joined: Tue Jul 02, 2019 2:28 pm

Re: compile Linux C programs to use cpu cores exclusively

Sat Aug 03, 2019 8:46 pm

If you don't want to use taskset and you want your programs to set their own CPU affinity then do this

Code: Select all

#define _GNU_SOURCE

#include <stdio.h>
#include <stdlib.h>
#include <sched.h>

int main()
{
  cpu_set_t cpus;

  CPU_ZERO(&cpus);

  CPU_SET(2, &cpus);
  CPU_SET(3, &cpus);

  if (sched_setaffinity(0, sizeof(cpus), &cpus)) {
    perror("sched_setaffinity");
    exit(1);
  }
  
  /* Do stuff */
}
You can check by looking at /proc/<PID of your program>/status and looking at the Cpus_allowed_list line.

dsyleixa123
Posts: 405
Joined: Mon Jun 11, 2018 11:22 am

Re: compile Linux C programs to use cpu cores exclusively

Sun Aug 04, 2019 7:33 am

that looks really cool, thank you very much!

dsyleixa123
Posts: 405
Joined: Mon Jun 11, 2018 11:22 am

Re: compile Linux C programs to use cpu cores exclusively

Sun Aug 04, 2019 2:25 pm

just to be sure:
Even for that cpu_set_t cpus approach I still have to set in config.txt
isolcpus=2,3
or don't I?

ejolson
Posts: 3578
Joined: Tue Mar 18, 2014 11:47 am

Re: compile Linux C programs to use cpu cores exclusively

Sun Aug 04, 2019 2:53 pm

dsyleixa123 wrote:
Sun Aug 04, 2019 2:25 pm
just to be sure:
Even for that cpu_set_t cpus approach I still have to set in config.txt
isolcpus=2,3
or don't I?
With systemd things may have changed or may change tomorrow. It's worth adding the isocpus line and then checking it is enough to prevent system tasks from being scheduled on those processors.

Return to “C/C++”