Kradenko
Posts: 12
Joined: Fri Jan 01, 2016 10:08 am

Raspbian refuses to use Swap Space

Tue Oct 01, 2019 4:52 pm

Hi all,

I have been using the new Raspbian Buster OS on a Raspberry Pi 4 with 4GB of memory. I have also added a 750GB external HDD to the PI and placed a 8GB swap file on this external harddrive. Below is my configuration as per multiple tutorials and posts online.

Code: Select all

# /etc/dphys-swapfile - user settings for dphys-swapfile package
# author Neil Franklin, last modification 2010.05.05
# copyright ETH Zuerich Physics Departement
#   use under either modified/non-advertising BSD or GPL license

# this file is sourced with . so full normal sh syntax applies

# the default settings are added as commented out CONF_*=* lines


# where we want the swapfile to be, this is the default
CONF_SWAPFILE=/mnt/external/swapfile

# set size to absolute value, leaving empty (default) then uses computed value
#   you most likely don't want this, unless you have an special disk situation
CONF_SWAPSIZE=8192

# set size to computed value, this times RAM size, dynamically adapts,
#   guarantees that there is enough swap without wasting disk space on excess
#CONF_SWAPFACTOR=2

# restrict size (computed and absolute!) to maximally this limit
#   can be set to empty for no limit, but beware of filled partitions!
#   this is/was a (outdated?) 32bit kernel limit (in MBytes), do not overrun it
#   but is also sensible on 64bit to prevent filling /var or even / partition
CONF_MAXSWAP=8192

Code: Select all

pi@raspberrypi:~/src $ free -h
              total        used        free      shared  buff/cache   available
Mem:          3.9Gi       235Mi       377Mi       147Mi       3.3Gi       3.4Gi
Swap:         8.0Gi       3.0Mi       8.0Gi

Also when I look at "top" it doesn't ever go into this swap space.

Code: Select all

top - 17:49:01 up  1:44,  3 users,  load average: 0.04, 0.68, 2.49
Tasks: 134 total,   1 running, 133 sleeping,   0 stopped,   0 zombie
%Cpu0  :  1.0 us,  0.0 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  1.0 us,  1.0 sy,  0.0 ni, 98.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  1.0 us,  1.9 sy,  0.0 ni, 97.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   3955.6 total,    377.4 free,    235.1 used,   3343.1 buff/cache
MiB Swap:   8192.0 total,   8188.5 free,      3.5 used.   3431.3 avail Mem

All processing and large compilations fail with a "out of memory error" but it never uses the swap space. Does anyone have an idea why it won't allow me to use any of the swap space?

Any suggestions to resolve this will be much appreciated.

jj_0
Posts: 68
Joined: Wed Jul 11, 2012 7:07 am

Re: Raspbian refuses to use Swap Space

Tue Oct 01, 2019 5:52 pm

Raspbian uses a different mechanism for configuring the swapfile.

Kradenko
Posts: 12
Joined: Fri Jan 01, 2016 10:08 am

Re: Raspbian refuses to use Swap Space

Tue Oct 01, 2019 5:56 pm

jj_0 wrote:
Tue Oct 01, 2019 5:52 pm
Raspbian uses a different mechanism for configuring the swapfile.

Tried that. It allocates the disk space and shows it as swap space. But never uses it. Wondering if it's maybe a kernel or software related issue?

jj_0
Posts: 68
Joined: Wed Jul 11, 2012 7:07 am

Re: Raspbian refuses to use Swap Space

Tue Oct 01, 2019 6:14 pm

Hmmm. Can you show the output of:

Code: Select all

 systemctl status dphys-swapfile

Maybe you need to remove the entries you put in fstab?

Kradenko
Posts: 12
Joined: Fri Jan 01, 2016 10:08 am

Re: Raspbian refuses to use Swap Space

Tue Oct 01, 2019 7:04 pm

jj_0 wrote:
Tue Oct 01, 2019 6:14 pm
Hmmm. Can you show the output of:

Code: Select all

 systemctl status dphys-swapfile

Maybe you need to remove the entries you put in fstab?

Here is the output:

Code: Select all

pi@raspberrypi:~ $  systemctl status dphys-swapfile
● dphys-swapfile.service - dphys-swapfile - set up, mount/unmount, and delete a swap file
   Loaded: loaded (/lib/systemd/system/dphys-swapfile.service; enabled; vendor preset: enabled)
   Active: active (exited) since Tue 2019-10-01 16:04:00 BST; 3h 58min ago
     Docs: man:dphys-swapfile(8)
  Process: 332 ExecStart=/sbin/dphys-swapfile setup (code=exited, status=0/SUCCESS)
  Process: 410 ExecStart=/sbin/dphys-swapfile swapon (code=exited, status=0/SUCCESS)
 Main PID: 410 (code=exited, status=0/SUCCESS)

Oct 01 16:03:59 raspberrypi systemd[1]: Starting dphys-swapfile - set up, mount/unmount, and delete a swap file...
Oct 01 16:03:59 raspberrypi dphys-swapfile[332]: want /mnt/external/swapfile=8192MByte, checking existing: deleting wrong size file (2147483648), generating swapfile ... of 8192MBytes
Oct 01 16:04:00 raspberrypi systemd[1]: Started dphys-swapfile - set up, mount/unmount, and delete a swap file.

Also, here is my fstab:

Code: Select all

pi@raspberrypi:~ $ cat /etc/fstab
proc            /proc           proc    defaults          0       0
PARTUUID=4710f635-01  /boot           vfat    defaults          0       2
PARTUUID=4710f635-02  /               ext4    defaults,noatime  0       1
/dev/sda1       /mnt/external   ext4    auto,nofail,user,noatime        0       0
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that

jj_0
Posts: 68
Joined: Wed Jul 11, 2012 7:07 am

Re: Raspbian refuses to use Swap Space

Tue Oct 01, 2019 7:09 pm

Sorry, that's as far as I can imagine.

logcabin
Posts: 13
Joined: Sat Jul 20, 2019 1:18 am

Re: Raspbian refuses to use Swap Space

Tue Oct 01, 2019 9:17 pm

Maybe the swap file is too large. For test purposes, try reducing the size to 2 GB.

LTolledo
Posts: 1959
Joined: Sat Mar 17, 2018 7:29 am
Location: Anime Heartland

Re: Raspbian refuses to use Swap Space

Tue Oct 01, 2019 9:51 pm

depending on how you "load-up" applications on your RPi4B-4G it may never use the swap file if usage is "normal desktop use" as there are plenty of RAM for the system to "play around"
that's why the it was recommended before to keep the swap-file deactivated.

you can verify its use, as well as the RAM use by running

Code: Select all

htop
is there a special reason why you need to enable swap on your system?
"Don't come to me with 'issues' for I don't know how to deal with those
Come to me with 'problems' and I'll help you find solutions"

Some people be like:
"Help me! Am drowning! But dont you dare touch me nor come near me!"

Kradenko
Posts: 12
Joined: Fri Jan 01, 2016 10:08 am

Re: Raspbian refuses to use Swap Space

Tue Oct 01, 2019 10:14 pm

LTolledo wrote:
Tue Oct 01, 2019 9:51 pm
depending on how you "load-up" applications on your RPi4B-4G it may never use the swap file if usage is "normal desktop use" as there are plenty of RAM for the system to "play around"
that's why the it was recommended before to keep the swap-file deactivated.

you can verify its use, as well as the RAM use by running

Code: Select all

htop
is there a special reason why you need to enable swap on your system?

htop is an awesome tool! Wow, didn't know it existed. Thnx!

There are two things I am trying to do on the Raspberry Pi.

1. I am trying to import map data into a postgres database. Here is the writeup about it https://switch2osm.org/manually-buildin ... 18-04-lts/. It am not trying to import the entire globe at least. Just a medium size area.
2. I am trying to generate the data files for the open source routing machine. This can be found here https://github.com/Project-OSRM/osrm-ba ... nning-OSRM.

So it's not day to day type activities. It is big data crunching. Really strange that even though the swap space is active, it just refuses to use the swap space.

Kradenko
Posts: 12
Joined: Fri Jan 01, 2016 10:08 am

Re: Raspbian refuses to use Swap Space

Tue Oct 01, 2019 10:26 pm

Just to add to this. Is there something Rasbian that prevents apps from using more than X amount of ram? maybe a page limit of sorts?

I tried running it now again and it bombed out with the following output:

Code: Select all

pi@raspberrypi:/mnt/external/maps $ osrm-extract -p ~/osrm-backend/profiles/car.lua africa-latest.osm.pbf
[info] Parsed 0 location-dependent features with 0 GeoJSON polygons
[info] Using script /home/pi/osrm-backend/profiles/car.lua
[info] Input file: africa-latest.osm.pbf
[info] Profile: car.lua
[info] Threads: 4
[info] Parsing in progress..
[info] input file generated by osmium/1.8.0
[info] timestamp: 2019-09-30T20:18:02Z
[info] Using profile api version 4
[info] Found 3 turn restriction tags:
[info]   motorcar
[info]   motor_vehicle
[info]   vehicle
[info] Parse relations ...
[info] Parse ways and nodes ...
[info] Using profile api version 4
[info] Using profile api version 4
[info] Using profile api version 4
[info] RAM: peak bytes used: 1193299968
[error] [exception] std::bad_alloc
[error] Please provide more memory or consider using a larger swapfile

That says that it uses no more than 1193,3 MB. Could this be a software issue? Just strange that all these functions bomb out on such low consumption.

Thanks again for the assistance!

laurent
Posts: 306
Joined: Thu Jul 26, 2012 11:24 am

Re: Raspbian refuses to use Swap Space

Wed Oct 02, 2019 1:17 pm

Hi,
Its likely to be a maximum memory per process limitation. Even with many gigabytes of swap, you can't allocate more than a limited amount of RAM per process. But even with a 32 bit OS, it must be 4GB so I can't explain why you are limited to 1GB or so.

Kradenko
Posts: 12
Joined: Fri Jan 01, 2016 10:08 am

Re: Raspbian refuses to use Swap Space

Wed Oct 02, 2019 1:43 pm

laurent wrote:
Wed Oct 02, 2019 1:17 pm
Hi,
Its likely to be a maximum memory per process limitation. Even with many gigabytes of swap, you can't allocate more than a limited amount of RAM per process. But even with a 32 bit OS, it must be 4GB so I can't explain why you are limited to 1GB or so.

I have two ways I guess I could try see what causes it.

1. Try the new 64Bit Kernel for Raspbian
2. Setup a Ubuntu VM with 4GB of ram with a 8GB swap file. See if the process runs on a normal x86 system using swap space.

Will try tonight and report back.

Kradenko
Posts: 12
Joined: Fri Jan 01, 2016 10:08 am

Re: Raspbian refuses to use Swap Space

Thu Oct 03, 2019 11:23 am

Kradenko wrote:
Wed Oct 02, 2019 1:43 pm
laurent wrote:
Wed Oct 02, 2019 1:17 pm
Hi,
Its likely to be a maximum memory per process limitation. Even with many gigabytes of swap, you can't allocate more than a limited amount of RAM per process. But even with a 32 bit OS, it must be 4GB so I can't explain why you are limited to 1GB or so.

I have two ways I guess I could try see what causes it.

1. Try the new 64Bit Kernel for Raspbian
2. Setup a Ubuntu VM with 4GB of ram with a 8GB swap file. See if the process runs on a normal x86 system using swap space.

Will try tonight and report back.

Ok, so setting the kernel to 64Bit made no difference. As for the VM test on Ubuntu it worked flawlessly. It allocated swap memory as needed on exactly the same specs as the Pi 4 with 4GB of RAM and 8GB of swap.
memory-2.jpg
memory-2.jpg (150.19 KiB) Viewed 626 times

What I did notice while watching the htop as LTolledo had suggested, was the VIRT column would not allocate more that 1280MB of memory. Then it runs out of memory. So I am wondering now if there is a limit set by the OS which doesn't allow more than 1280MB of memory to be allocated to a process? Could this be?

dickon
Posts: 482
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, just outside Reading

Re: Raspbian refuses to use Swap Space

Thu Oct 03, 2019 11:48 am

Kradenko wrote:
Wed Oct 02, 2019 1:43 pm
2. Setup a Ubuntu VM with 4GB of ram with a 8GB swap file. See if the process runs on a normal x86 system using swap space.
x86, or amd64? If the latter, then you're hitting per-process limits on the Pi; given it's a database, it may not be actually using the memory, but may be using the address space.

I'd be tempted to boot the 64b kernel with a 64b Debian userland (in a chroot is fine) and see if that fixes it; I suspect it will. I think you've hit one of those niche use-cases where a full 64b OS is useful.

Kradenko
Posts: 12
Joined: Fri Jan 01, 2016 10:08 am

Re: Raspbian refuses to use Swap Space

Thu Oct 03, 2019 12:00 pm

dickon wrote:
Thu Oct 03, 2019 11:48 am
Kradenko wrote:
Wed Oct 02, 2019 1:43 pm
2. Setup a Ubuntu VM with 4GB of ram with a 8GB swap file. See if the process runs on a normal x86 system using swap space.
x86, or amd64? If the latter, then you're hitting per-process limits on the Pi; given it's a database, it may not be actually using the memory, but may be using the address space.

I'd be tempted to boot the 64b kernel with a 64b Debian userland (in a chroot is fine) and see if that fixes it; I suspect it will. I think you've hit one of those niche use-cases where a full 64b OS is useful.

The Ubuntu in the VM that I tested is amd64. So you might be right there. Will give it a go and see. Thanks!

tului
Posts: 12
Joined: Sat Nov 09, 2013 6:19 am

Re: Raspbian refuses to use Swap Space

Sat Oct 12, 2019 7:14 am

What's the memory split on 32 bit ARMfor apps/kernel

Kradenko
Posts: 12
Joined: Fri Jan 01, 2016 10:08 am

Re: Raspbian refuses to use Swap Space

Sun Oct 13, 2019 10:02 am

Well, I have now installed Gentoo 64Bit on the Pi and it has gone a lot better. I can now actually use the full 4GB of memory. It also allows me to go deep into swap space. 8GB on an external harddrive. Still bombs on larger datasets.

Will wait for the official 64Bit launch of Raspbian and try again.

Thanks for the assistance!

tului
Posts: 12
Joined: Sat Nov 09, 2013 6:19 am

Re: Raspbian refuses to use Swap Space

Tue Oct 15, 2019 10:05 am

Kradenko wrote:
Sun Oct 13, 2019 10:02 am
Well, I have now installed Gentoo 64Bit on the Pi and it has gone a lot better. I can now actually use the full 4GB of memory. It also allows me to go deep into swap space. 8GB on an external harddrive. Still bombs on larger datasets.

Will wait for the official 64Bit launch of Raspbian and try again.

Thanks for the assistance!
Afaik it's official just in testing.

Return to “Advanced users”