tonyhain
Posts: 7
Joined: Sat Feb 23, 2019 5:23 pm

Re: Moving Linux kernel to 4.19

Tue Feb 26, 2019 9:43 pm

Didn't see the comment about stale hostapd as I was putting my previous comment together. After seeing that though I did a quick check and found the one I was using is even older

Code: Select all

# hostapd -v 
hostapd v2.3
User space daemon for IEEE 802.11 AP management,
IEEE 802.1X/WPA/WPA2/EAP/RADIUS Authenticator
Copyright (c) 2002-2014, Jouni Malinen <j@w1.fi> and contributors
will contact cyoung about updating it. Since the stratux distribution is still 4.9, it is unlikely hostapd will get updated on it until AP mode is stable for more than a couple of weeks on either 4.14 or 4.19.

In the FWIW department, after 1 hr 5 min uptime, the AP mode dropped in 4.14.98 just like it had been in 4.9

Code: Select all

[ 3935.209690] brcmfmac: brcmf_sdio_hostmail: Unknown mailbox data content: 0x40012
[ 4007.328004] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[ 4009.887982] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[ 4009.887999] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110
[ 4140.768549] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[ 4140.768564] brcmfmac: brcmf_cfg80211_get_channel: chanspec failed (-110)
[ 4143.328516] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[ 4143.328532] brcmfmac: brcmf_cfg80211_get_tx_power: error (-110)
[ 4212.368820] brcmfmac: brcmf_netdev_wait_pend8021x: Timed out waiting for no pending 802.1x packets
modprobe bounce had no effect

Code: Select all

[ 4484.799515] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing to do.
[ 4484.799522] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing to do.
[ 4484.799645] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing to do.
[ 4484.810414] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing to do.
[ 4484.810430] brcmfmac: brcmf_cfg80211_get_channel: chanspec failed (-5)
[ 4484.991871] usbcore: deregistering interface driver brcmfmac
[ 4488.815263] brcmfmac: F1 signature read @0x18000000=0x1541a9a6
[ 4488.823261] brcmfmac: brcmf_fw_map_chip_to_name: using brcm/brcmfmac43430-sdio.bin for chip 0x00a9a6(43430) rev 0x000001
[ 4488.823570] usbcore: registered new interface driver brcmfmac
[ 4491.489439] brcmfmac: brcmf_sdio_bus_rxctl: resumed on timeout
[ 4491.492721] brcmfmac: brcmf_c_get_clm_name: retrieving revision info failed (-110)
[ 4491.492730] brcmfmac: brcmf_c_process_clm_blob: get CLM blob file name failed (-110)
[ 4491.492738] brcmfmac: brcmf_c_preinit_dcmds: download CLM blob file failed, -110
[ 4491.492745] brcmfmac: brcmf_bus_started: failed: -110
[ 4491.492767] brcmfmac: brcmf_sdio_firmware_callback: dongle is not responding

Code: Select all

# iw phy
# echo -n 3f300000.mmc > /sys/devices/platform/soc/3f300000.mmc/driver/unbind
# echo -n 3f300000.mmc > /sys/bus/platform/drivers/mmc-bcm2835/bind
# iw phy
# dmesg
...
[ 5039.299639] ------------[ cut here ]------------
[ 5039.299680] WARNING: CPU: 3 PID: 984 at kernel/irq/manage.c:1542 __free_irq+0xd0/0x31c
[ 5039.299689] Trying to free already-free IRQ 92
[ 5039.299696] Modules linked in: brcmfmac brcmutil cfg80211 rfkill ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables x_tables cdc_acm snd_bcm2835(C) snd_pcm snd_timer snd i2c_bcm2835 joydev rpi_backlight uio_pdrv_genirq rpi_ft5406 uio fixed evdev i2c_dev i2c_bcm2708 ipv6 [last unloaded: brcmutil]
[ 5039.299933] CPU: 3 PID: 984 Comm: bash Tainted: G        WC      4.14.98-v7+ #1200
[ 5039.299943] Hardware name: BCM2835
[ 5039.299984] [<8010ff30>] (unwind_backtrace) from [<8010c174>] (show_stack+0x20/0x24)
[ 5039.300017] [<8010c174>] (show_stack) from [<8078cf04>] (dump_stack+0xd4/0x118)
[ 5039.300039] [<8078cf04>] (dump_stack) from [<8011dd70>] (__warn+0xf8/0x110)
[ 5039.300057] [<8011dd70>] (__warn) from [<8011ddd0>] (warn_slowpath_fmt+0x48/0x50)
[ 5039.300079] [<8011ddd0>] (warn_slowpath_fmt) from [<801773ac>] (__free_irq+0xd0/0x31c)
[ 5039.300114] [<801773ac>] (__free_irq) from [<8017769c>] (free_irq+0x4c/0x94)
[ 5039.300148] [<8017769c>] (free_irq) from [<8017b554>] (devm_irq_release+0x1c/0x20)
[ 5039.300177] [<8017b554>] (devm_irq_release) from [<8054b5b0>] (release_nodes+0x17c/0x1e4)
[ 5039.300215] [<8054b5b0>] (release_nodes) from [<8054c054>] (devres_release_all+0x40/0x5c)
[ 5039.300241] [<8054c054>] (devres_release_all) from [<8054860c>] (device_release_driver_internal+0x164/0x1ec)
[ 5039.300259] [<8054860c>] (device_release_driver_internal) from [<805486b4>] (device_release_driver+0x20/0x24)
[ 5039.300273] [<805486b4>] (device_release_driver) from [<80546a00>] (unbind_store+0x88/0x108)
[ 5039.300287] [<80546a00>] (unbind_store) from [<80545f60>] (drv_attr_store+0x30/0x3c)
[ 5039.300304] [<80545f60>] (drv_attr_store) from [<8030acc0>] (sysfs_kf_write+0x54/0x58)
[ 5039.300337] [<8030acc0>] (sysfs_kf_write) from [<8030a334>] (kernfs_fop_write+0xdc/0x1c0)
[ 5039.300360] [<8030a334>] (kernfs_fop_write) from [<8028bee4>] (__vfs_write+0x38/0x138)
[ 5039.300378] [<8028bee4>] (__vfs_write) from [<8028c1ac>] (vfs_write+0xb4/0x1bc)
[ 5039.300410] [<8028c1ac>] (vfs_write) from [<8028c3fc>] (SyS_write+0x54/0xb0)
[ 5039.300434] [<8028c3fc>] (SyS_write) from [<80108000>] (ret_fast_syscall+0x0/0x28)
[ 5039.300445] ---[ end trace 338168505c054132 ]---
[ 5041.554415] mmc-bcm2835 3f300000.mmc: mmc_debug:0 mmc_debug2:0
[ 5041.554428] mmc-bcm2835 3f300000.mmc: DMA channel allocated
[ 5041.623849] mmc1: queuing unknown CIS tuple 0x41 (4 bytes)
[ 5041.626028] mmc1: queuing unknown CIS tuple 0x04 (0 bytes)
[ 5041.626672] mmc1: bad CIS tuple 0x20 (0 bytes)
[ 5041.626689] mmc1: error -22 whilst initialising SDIO card
[ 5041.704652] mmc1: queuing unknown CIS tuple 0x41 (4 bytes)
[ 5041.707501] mmc1: queuing unknown CIS tuple 0x04 (0 bytes)
[ 5041.708316] mmc1: bad CIS tuple 0x20 (0 bytes)
[ 5041.708332] mmc1: error -22 whilst initialising SDIO card
[ 5041.798438] mmc1: queuing unknown CIS tuple 0x41 (4 bytes)
[ 5041.802761] mmc1: queuing unknown CIS tuple 0x04 (0 bytes)
[ 5041.803953] mmc1: bad CIS tuple 0x20 (0 bytes)
[ 5041.803970] mmc1: error -22 whilst initialising SDIO card
[ 5041.899864] mmc1: queuing unknown CIS tuple 0x41 (4 bytes)
[ 5041.907898] mmc1: queuing unknown CIS tuple 0x04 (0 bytes)
[ 5041.910194] mmc1: bad CIS tuple 0x20 (0 bytes)
[ 5041.910211] mmc1: error -22 whilst initialising SDIO card
This would appear to have nothing to do with hostapd as the device simply disappears out from under it and can't be reestablished. Will swap to the 7.45.41.46 sdio.bin and try again.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5341
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Moving Linux kernel to 4.19

Tue Feb 26, 2019 9:58 pm

swahren wrote:
Tue Feb 26, 2019 8:21 pm
Because of the many complains about broken AP mode, i want to link to my post from November:
https://www.raspberrypi.org/forums/view ... 5#p1396409

Since this won't be fixed in mainline, we have two options:
1) update hostapd to a more recent version (e.g. from Buster)
2) revert the mentioned commit
Thanks for reminder. I'll check with colleagues which option is preferred.
(Note: we're planning on switching raspbian to buster as soon as it's stable,
so hopefully this issue will vanish, but that is likely a few months off, so a temporary solution is wanted).

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5341
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Moving Linux kernel to 4.19

Tue Mar 05, 2019 3:25 pm

swahren wrote:
Tue Feb 26, 2019 8:21 pm
Since this won't be fixed in mainline, we have two options:
1) update hostapd to a more recent version (e.g. from Buster)
2) revert the mentioned commit
hostapd in raspbian has been updated to version from buster. Hopefully this resolved the issue.

lowflyer
Posts: 78
Joined: Sat Jun 01, 2013 2:27 pm

Re: Moving Linux kernel to 4.19

Wed Mar 06, 2019 10:00 am

@dom Thanks!!! You are a star!
dom wrote: hostapd in raspbian has been updated to version from buster. Hopefully this resolved the issue.
Starting with my working AP using 4.14.98-v7+ #1200, I did this...

Code: Select all

sudo apt update
sudo apt upgrade
sudo systemctl enable hostapd.service
sudo shutdown -r now
And the new version of hostapd (v2.6) works with the 4.14.98-v7+ #1200 kernel.

Then:

Code: Select all

sudo rpi-update
sudo shutdown -r now
And the AP works with kernel 4.19.25-v7+ #1205.

Well done!!

User avatar
kozman
Posts: 53
Joined: Tue Sep 11, 2018 3:40 pm

Re: Moving Linux kernel to 4.19

Wed Mar 06, 2019 2:42 pm

Good to hear the updated Hostpad version is helping.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5341
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Moving Linux kernel to 4.19

Wed Mar 06, 2019 3:25 pm

lowflyer wrote:
Wed Mar 06, 2019 10:00 am
And the AP works with kernel 4.19.25-v7+ #1205.
Good to hear!

lems
Posts: 2
Joined: Thu Mar 07, 2019 7:14 am

Re: Moving Linux kernel to 4.19

Thu Mar 07, 2019 7:23 am

dariuszb wrote:
Fri Feb 22, 2019 8:52 am
With external disk attached when upgraded to 4.19 e4defrag commands kills the system:

Code: Select all

[ 1192.086591] e4defrag invoked oom-killer: gfp_mask=0x6000d0(GFP_KERNEL|__GFP_RECLAIMABLE), nodemask=(null), order=0, oom_score_adj=0
[ 1192.086604] e4defrag cpuset=/ mems_allowed=0
[ 1192.086630] CPU: 2 PID: 2149 Comm: e4defrag Tainted: G         C O      4.19.23-v7+ #1203
[ 1192.086634] Hardware name: BCM2835
[ 1192.086668] [<80111ea8>] (unwind_backtrace) from [<8010d440>] (show_stack+0x20/0x24)
[ 1192.086685] [<8010d440>] (show_stack) from [<8080c880>] (dump_stack+0xd4/0x118)
[ 1192.086703] [<8080c880>] (dump_stack) from [<8023a7f0>] (dump_header+0x80/0x250)
[ 1192.086718] [<8023a7f0>] (dump_header) from [<80239b68>] (oom_kill_process+0x358/0x3a8)
[ 1192.086731] [<80239b68>] (oom_kill_process) from [<8023a498>] (out_of_memory+0x134/0x36c)
[ 1192.086746] [<8023a498>] (out_of_memory) from [<802408c0>] (__alloc_pages_nodemask+0x1024/0x1178)
[ 1192.086762] [<802408c0>] (__alloc_pages_nodemask) from [<802901d0>] (new_slab+0x520/0x6b4)
[ 1192.086775] [<802901d0>] (new_slab) from [<80292190>] (___slab_alloc.constprop.11+0x454/0x594)
[ 1192.086788] [<80292190>] (___slab_alloc.constprop.11) from [<80292314>] (__slab_alloc.constprop.10+0x44/0x90)
[ 1192.086801] [<80292314>] (__slab_alloc.constprop.10) from [<80292ae0>] (kmem_cache_alloc+0x20c/0x23c)
[ 1192.086815] [<80292ae0>] (kmem_cache_alloc) from [<802c3450>] (__d_alloc+0x34/0x1f0)
[ 1192.086830] [<802c3450>] (__d_alloc) from [<802c362c>] (d_alloc+0x20/0x88)
[ 1192.086843] [<802c362c>] (d_alloc) from [<802c3ab0>] (d_alloc_parallel+0x68/0x500)
[ 1192.086858] [<802c3ab0>] (d_alloc_parallel) from [<802b3fdc>] (__lookup_slow+0x6c/0x174)
[ 1192.086873] [<802b3fdc>] (__lookup_slow) from [<802b4124>] (lookup_slow+0x40/0x54)
[ 1192.086885] [<802b4124>] (lookup_slow) from [<802b6f08>] (walk_component+0x1f8/0x2f8)
[ 1192.086896] [<802b6f08>] (walk_component) from [<802b75b4>] (path_lookupat+0x78/0x214)
[ 1192.086908] [<802b75b4>] (path_lookupat) from [<802b90d4>] (filename_lookup+0xac/0x11c)
[ 1192.086919] [<802b90d4>] (filename_lookup) from [<802b9254>] (user_path_at_empty+0x50/0x58)
[ 1192.086931] [<802b9254>] (user_path_at_empty) from [<802ad820>] (vfs_statx+0x7c/0xe8)
[ 1192.086944] [<802ad820>] (vfs_statx) from [<802ae444>] (sys_fstatat64+0x40/0x70)
[ 1192.086958] [<802ae444>] (sys_fstatat64) from [<80101000>] (ret_fast_syscall+0x0/0x28)
[ 1192.086963] Exception stack(0xa604bfa8 to 0xa604bff0)
[ 1192.086974] bfa0:                   7e9b3388 7e9b1024 00000007 01cd7f8b 7e9b0f78 00000100
[ 1192.086984] bfc0: 7e9b3388 7e9b1024 7e9b0f78 00000147 7e9b3388 7e9b3388 7e9b33c0 7e9b344c
[ 1192.086991] bfe0: 0000001c 7e9b0f58 76e78288 76e74cc0
[ 1192.086997] Mem-Info:
[ 1192.087017] active_anon:13885 inactive_anon:2626 isolated_anon:0
[ 1192.087017]  active_file:369 inactive_file:448 isolated_file:0
[ 1192.087017]  unevictable:438 dirty:0 writeback:0 unstable:0
[ 1192.087017]  slab_reclaimable:211896 slab_unreclaimable:9973
[ 1192.087017]  mapped:1277 shmem:2670 pagetables:608 bounce:0
It has been working perfectly for years including kernel 4.14.98. External disk has fair amount of files - 2GB in total, 2.5 million files.
I experienced this, too. I'm running a RPi 1 B headless, with gpu_mem_512=16. Upgrading to 4.19 resulted in rsnapshot no longer being able to complete (I have two USB disks attached via an active hub, both formatted using ext4). The OOM killer would start killing all kinds of processes, including rsync. Rolling back to 4.14. This setup has been working flawlessly the last years (first on Void Linux, now on Devuan).

edit: I also use a 1GB swap file that is located on one of the disks.
Last edited by lems on Thu Mar 07, 2019 9:47 am, edited 1 time in total.

Ernst
Posts: 1245
Joined: Sat Feb 04, 2017 9:39 am
Location: Germany

Re: Moving Linux kernel to 4.19

Thu Mar 07, 2019 8:22 am

I have decided to give hostapd another chance and it looks very promising.
See https://www.raspberrypi.org/forums/view ... 6#p1438636
The road to insanity is paved with static ip addresses

muhviehstarr
Posts: 1
Joined: Sun Mar 10, 2019 8:42 pm

Re: Moving Linux kernel to 4.19

Sun Mar 10, 2019 8:45 pm

lems wrote:
Thu Mar 07, 2019 7:23 am
dariuszb wrote:
Fri Feb 22, 2019 8:52 am
With external disk attached when upgraded to 4.19 e4defrag commands kills the system:

Code: Select all

[ 1192.086591] e4defrag invoked oom-killer: gfp_mask=0x6000d0(GFP_KERNEL|__GFP_RECLAIMABLE), nodemask=(null), order=0, oom_score_adj=0
[ 1192.086604] e4defrag cpuset=/ mems_allowed=0
[ 1192.086630] CPU: 2 PID: 2149 Comm: e4defrag Tainted: G         C O      4.19.23-v7+ #1203
[ 1192.086634] Hardware name: BCM2835
[ 1192.086668] [<80111ea8>] (unwind_backtrace) from [<8010d440>] (show_stack+0x20/0x24)
[ 1192.086685] [<8010d440>] (show_stack) from [<8080c880>] (dump_stack+0xd4/0x118)
[ 1192.086703] [<8080c880>] (dump_stack) from [<8023a7f0>] (dump_header+0x80/0x250)
[ 1192.086718] [<8023a7f0>] (dump_header) from [<80239b68>] (oom_kill_process+0x358/0x3a8)
[ 1192.086731] [<80239b68>] (oom_kill_process) from [<8023a498>] (out_of_memory+0x134/0x36c)
[ 1192.086746] [<8023a498>] (out_of_memory) from [<802408c0>] (__alloc_pages_nodemask+0x1024/0x1178)
[ 1192.086762] [<802408c0>] (__alloc_pages_nodemask) from [<802901d0>] (new_slab+0x520/0x6b4)
[ 1192.086775] [<802901d0>] (new_slab) from [<80292190>] (___slab_alloc.constprop.11+0x454/0x594)
[ 1192.086788] [<80292190>] (___slab_alloc.constprop.11) from [<80292314>] (__slab_alloc.constprop.10+0x44/0x90)
[ 1192.086801] [<80292314>] (__slab_alloc.constprop.10) from [<80292ae0>] (kmem_cache_alloc+0x20c/0x23c)
[ 1192.086815] [<80292ae0>] (kmem_cache_alloc) from [<802c3450>] (__d_alloc+0x34/0x1f0)
[ 1192.086830] [<802c3450>] (__d_alloc) from [<802c362c>] (d_alloc+0x20/0x88)
[ 1192.086843] [<802c362c>] (d_alloc) from [<802c3ab0>] (d_alloc_parallel+0x68/0x500)
[ 1192.086858] [<802c3ab0>] (d_alloc_parallel) from [<802b3fdc>] (__lookup_slow+0x6c/0x174)
[ 1192.086873] [<802b3fdc>] (__lookup_slow) from [<802b4124>] (lookup_slow+0x40/0x54)
[ 1192.086885] [<802b4124>] (lookup_slow) from [<802b6f08>] (walk_component+0x1f8/0x2f8)
[ 1192.086896] [<802b6f08>] (walk_component) from [<802b75b4>] (path_lookupat+0x78/0x214)
[ 1192.086908] [<802b75b4>] (path_lookupat) from [<802b90d4>] (filename_lookup+0xac/0x11c)
[ 1192.086919] [<802b90d4>] (filename_lookup) from [<802b9254>] (user_path_at_empty+0x50/0x58)
[ 1192.086931] [<802b9254>] (user_path_at_empty) from [<802ad820>] (vfs_statx+0x7c/0xe8)
[ 1192.086944] [<802ad820>] (vfs_statx) from [<802ae444>] (sys_fstatat64+0x40/0x70)
[ 1192.086958] [<802ae444>] (sys_fstatat64) from [<80101000>] (ret_fast_syscall+0x0/0x28)
[ 1192.086963] Exception stack(0xa604bfa8 to 0xa604bff0)
[ 1192.086974] bfa0:                   7e9b3388 7e9b1024 00000007 01cd7f8b 7e9b0f78 00000100
[ 1192.086984] bfc0: 7e9b3388 7e9b1024 7e9b0f78 00000147 7e9b3388 7e9b3388 7e9b33c0 7e9b344c
[ 1192.086991] bfe0: 0000001c 7e9b0f58 76e78288 76e74cc0
[ 1192.086997] Mem-Info:
[ 1192.087017] active_anon:13885 inactive_anon:2626 isolated_anon:0
[ 1192.087017]  active_file:369 inactive_file:448 isolated_file:0
[ 1192.087017]  unevictable:438 dirty:0 writeback:0 unstable:0
[ 1192.087017]  slab_reclaimable:211896 slab_unreclaimable:9973
[ 1192.087017]  mapped:1277 shmem:2670 pagetables:608 bounce:0
It has been working perfectly for years including kernel 4.14.98. External disk has fair amount of files - 2GB in total, 2.5 million files.
I experienced this, too. I'm running a RPi 1 B headless, with gpu_mem_512=16. Upgrading to 4.19 resulted in rsnapshot no longer being able to complete (I have two USB disks attached via an active hub, both formatted using ext4). The OOM killer would start killing all kinds of processes, including rsync. Rolling back to 4.14. This setup has been working flawlessly the last years (first on Void Linux, now on Devuan).

edit: I also use a 1GB swap file that is located on one of the disks.

Tested this on rpi3b+ with (custom) 4.19 kernel under archlinux

after a minute the pi freezed, had to reboot via power cable

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5341
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Moving Linux kernel to 4.19

Tue Mar 12, 2019 12:19 pm

dariuszb wrote:
Fri Feb 22, 2019 8:52 am
With external disk attached when upgraded to 4.19 e4defrag commands kills the system:
Possibly this issue: https://github.com/raspberrypi/linux/issues/2829
It's an upstream issue and solutions are being discussed there.
We'll pick up any fixes when they are decided on.

dariuszb
Posts: 33
Joined: Sun Feb 21, 2016 3:55 pm

Re: Moving Linux kernel to 4.19

Tue Mar 12, 2019 6:16 pm

dom wrote:
Tue Mar 12, 2019 12:19 pm
dariuszb wrote:
Fri Feb 22, 2019 8:52 am
With external disk attached when upgraded to 4.19 e4defrag commands kills the system:
Possibly this issue: https://github.com/raspberrypi/linux/issues/2829
It's an upstream issue and solutions are being discussed there.
We'll pick up any fixes when they are decided on.
Thank you. It makes a lot of sense now. Since 4.19 I have multiple out of memory situations when dealing with ext4 eternal drive operations (it all worked in 4.14).Not only e4defrag but also simple operations like du or tar as soon as applied to folders with many entries - I realize that there are always some limits and 1GB of RAM is going to hit them rather early but this was a bit too early:)

User avatar
kozman
Posts: 53
Joined: Tue Sep 11, 2018 3:40 pm

Re: Moving Linux kernel to 4.19

Wed Mar 13, 2019 2:52 pm

Phoronix did a benchmark of 4.14 against 4.19 (https://www.phoronix.com/scan.php?page= ... -419&num=1). Surprisingly, 4.19 is not substantially more performant. :cry: Too bad. Here's hoping later 4.19.x kernels and a final Buster can regain *some* performance in the future.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23876
Joined: Sat Jul 30, 2011 7:41 pm

Re: Moving Linux kernel to 4.19

Wed Mar 13, 2019 4:12 pm

kozman wrote:
Wed Mar 13, 2019 2:52 pm
Phoronix did a benchmark of 4.14 against 4.19 (https://www.phoronix.com/scan.php?page= ... -419&num=1). Surprisingly, 4.19 is not substantially more performant. :cry: Too bad. Here's hoping later 4.19.x kernels and a final Buster can regain *some* performance in the future.
Why would you expect noticeably more performance? The Linux kernel is pretty well optimised already, so releases are unlikely to provide much in the way of improved oomph. In fact, given the increased emphasis on security, that might require decreases in performance due to extra code requried in critical paths.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

User avatar
kozman
Posts: 53
Joined: Tue Sep 11, 2018 3:40 pm

Re: Moving Linux kernel to 4.19

Wed Mar 13, 2019 4:47 pm

jamesh wrote:
Wed Mar 13, 2019 4:12 pm
kozman wrote:
Wed Mar 13, 2019 2:52 pm
Phoronix did a benchmark of 4.14 against 4.19 (https://www.phoronix.com/scan.php?page= ... -419&num=1). Surprisingly, 4.19 is not substantially more performant. :cry: Too bad. Here's hoping later 4.19.x kernels and a final Buster can regain *some* performance in the future.
Why would you expect noticeably more performance? The Linux kernel is pretty well optimised already, so releases are unlikely to provide much in the way of improved oomph. In fact, given the increased emphasis on security, that might require decreases in performance due to extra code requried in critical paths.
Did I not read somewhere that the 4.18 kernel is where *full* and *complete* Pi 3/3+ support came about? I would have surmised that it would have made more than some difference in performance. Only some Spectre stuff affected Arm and in only the most minor of ways. Most of the mitigation perf drops came from non-Arch specific fixes in the kernel. So while I did expect some perf loss going from 4.14 to 4.19, some perf drops were quite dramatic in what the Phoronix benchmarks showed. Which is why it should be interesting to see what Buster will do against Stretch once it goes gold.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7415
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Moving Linux kernel to 4.19

Wed Mar 13, 2019 5:03 pm

kozman wrote:
Wed Mar 13, 2019 4:47 pm
Did I not read somewhere that the 4.18 kernel is where *full* and *complete* Pi 3/3+ support came about? I would have surmised that it would have made more than some difference in performance.
Iff you run the mainline kernel then this would be important. The majority of people don't, therefore it makes almost no difference at all.

Mainline is missing a fair number of the features that are present in the Pi downstream kernel, such as the FIQ handling for the USB controller.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
kozman
Posts: 53
Joined: Tue Sep 11, 2018 3:40 pm

Re: Moving Linux kernel to 4.19

Wed Mar 13, 2019 5:05 pm

6by9 wrote:
Wed Mar 13, 2019 5:03 pm
kozman wrote:
Wed Mar 13, 2019 4:47 pm
Did I not read somewhere that the 4.18 kernel is where *full* and *complete* Pi 3/3+ support came about? I would have surmised that it would have made more than some difference in performance.
Iff you run the mainline kernel then this would be important. The majority of people don't, therefore it makes almost no difference at all.

Mainline is missing a fair number of the features that are present in the Pi downstream kernel, such as the FIQ handling for the USB controller.
Gotcha.

dariuszb
Posts: 33
Joined: Sun Feb 21, 2016 3:55 pm

Re: Moving Linux kernel to 4.19

Thu Mar 21, 2019 10:58 am

dom wrote:
Tue Mar 12, 2019 12:19 pm
dariuszb wrote:
Fri Feb 22, 2019 8:52 am
With external disk attached when upgraded to 4.19 e4defrag commands kills the system:
Possibly this issue: https://github.com/raspberrypi/linux/issues/2829
It's an upstream issue and solutions are being discussed there.
We'll pick up any fixes when they are decided on.
Still not fixed in 4.19.29. Memory leak in caches always kills the system. Memory leaks during file operations so it happens faster when user performs some heavy file system jobs like people mentioned earlier on this forum.

User avatar
kozman
Posts: 53
Joined: Tue Sep 11, 2018 3:40 pm

Re: Moving Linux kernel to 4.19

Thu Mar 21, 2019 6:03 pm

dariuszb wrote:
Thu Mar 21, 2019 10:58 am
dom wrote:
Tue Mar 12, 2019 12:19 pm
dariuszb wrote:
Fri Feb 22, 2019 8:52 am
With external disk attached when upgraded to 4.19 e4defrag commands kills the system:
Possibly this issue: https://github.com/raspberrypi/linux/issues/2829
It's an upstream issue and solutions are being discussed there.
We'll pick up any fixes when they are decided on.
Still not fixed in 4.19.29. Memory leak in caches always kills the system. Memory leaks during file operations so it happens faster when user performs some heavy file system jobs like people mentioned earlier on this forum.

From what I can see in 4.19.30 relnotes, no ext4 fixes went in that might fix this either. It's looking like what's been talked about here (https://lkml.org/lkml/2019/1/30/28) and the patch here (https://lkml.org/lkml/2019/1/28/1865) are what will eventually be a fix. It's from late January so who knows how long it will take before it makes it into a future revision.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23876
Joined: Sat Jul 30, 2011 7:41 pm

Re: Moving Linux kernel to 4.19

Thu Mar 21, 2019 10:03 pm

kozman wrote:
Thu Mar 21, 2019 6:03 pm
dariuszb wrote:
Thu Mar 21, 2019 10:58 am
dom wrote:
Tue Mar 12, 2019 12:19 pm


Possibly this issue: https://github.com/raspberrypi/linux/issues/2829
It's an upstream issue and solutions are being discussed there.
We'll pick up any fixes when they are decided on.
Still not fixed in 4.19.29. Memory leak in caches always kills the system. Memory leaks during file operations so it happens faster when user performs some heavy file system jobs like people mentioned earlier on this forum.

From what I can see in 4.19.30 relnotes, no ext4 fixes went in that might fix this either. It's looking like what's been talked about here (https://lkml.org/lkml/2019/1/30/28) and the patch here (https://lkml.org/lkml/2019/1/28/1865) are what will eventually be a fix. It's from late January so who knows how long it will take before it makes it into a future revision.
We are currently discussing what to do with this internally. Upstream are pretty slow at fixing this, although there are possible patches that we may take and test.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

User avatar
kozman
Posts: 53
Joined: Tue Sep 11, 2018 3:40 pm

Re: Moving Linux kernel to 4.19

Thu Mar 21, 2019 10:10 pm

jamesh wrote:
Thu Mar 21, 2019 10:03 pm
kozman wrote:
Thu Mar 21, 2019 6:03 pm
dariuszb wrote:
Thu Mar 21, 2019 10:58 am


Still not fixed in 4.19.29. Memory leak in caches always kills the system. Memory leaks during file operations so it happens faster when user performs some heavy file system jobs like people mentioned earlier on this forum.

From what I can see in 4.19.30 relnotes, no ext4 fixes went in that might fix this either. It's looking like what's been talked about here (https://lkml.org/lkml/2019/1/30/28) and the patch here (https://lkml.org/lkml/2019/1/28/1865) are what will eventually be a fix. It's from late January so who knows how long it will take before it makes it into a future revision.
We are currently discussing what to do with this internally. Upstream are pretty slow at fixing this, although there are possible patches that we may take and test.

"Pretty slow" can sometimes be glacial with upstream adoption. I *totally* know what you mean. Months and years have passed by in the past. But seeing as how this affects quite a few moving parts, perhaps a fire will be lit under someone's posterior to get it done. Godspeed to you guys.

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6031
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: Moving Linux kernel to 4.19

Thu Mar 21, 2019 10:27 pm

I've tried that last patch and it didn't fix the problem.

A lot of the other proposed patches floating around on the mailing list have either already applied. One of them was applied upstream, then churned a bit and then reverted. If anybody could link me to some other patches or commits, I could give them a go.

User avatar
kozman
Posts: 53
Joined: Tue Sep 11, 2018 3:40 pm

Re: Moving Linux kernel to 4.19

Sat Mar 23, 2019 7:22 pm

ShiftPlusOne wrote:
Thu Mar 21, 2019 10:27 pm
I've tried that last patch and it didn't fix the problem.

A lot of the other proposed patches floating around on the mailing list have either already applied. One of them was applied upstream, then churned a bit and then reverted. If anybody could link me to some other patches or commits, I could give them a go.

I was taking a cursory glance at what's going into the upcoming 4.19.31 and there is one substantial ext4 change targeting inode stuff. Another ext4 change in there as well but seems to be related to "block" stuff. Not optimistic but but hopefully it has some benefit.

dariuszb
Posts: 33
Joined: Sun Feb 21, 2016 3:55 pm

Re: Moving Linux kernel to 4.19

Sat Mar 23, 2019 7:35 pm

kozman wrote:
Sat Mar 23, 2019 7:22 pm
ShiftPlusOne wrote:
Thu Mar 21, 2019 10:27 pm
I've tried that last patch and it didn't fix the problem.

A lot of the other proposed patches floating around on the mailing list have either already applied. One of them was applied upstream, then churned a bit and then reverted. If anybody could link me to some other patches or commits, I could give them a go.

I was taking a cursory glance at what's going into the upcoming 4.19.31 and there is one substantial ext4 change targeting inode stuff. Another ext4 change in there as well but seems to be related to "block" stuff. Not optimistic but but hopefully it has some benefit.
From my tests I can see that it is not per se ext4 issue but memory management of slab objects storing caches. All memory caches are affected I think, just ones related to ext4 operations are the easiest to fill in.

Having said that and anticipating that it might take some time to fix it wouldn't make sense to carry on with 4.14? The latest Pi release was 4.14.98 and it is already on 4.14.107

User avatar
kozman
Posts: 53
Joined: Tue Sep 11, 2018 3:40 pm

Re: Moving Linux kernel to 4.19

Sun Mar 24, 2019 1:12 am

dariuszb wrote:
Sat Mar 23, 2019 7:35 pm
kozman wrote:
Sat Mar 23, 2019 7:22 pm
ShiftPlusOne wrote:
Thu Mar 21, 2019 10:27 pm
I've tried that last patch and it didn't fix the problem.

A lot of the other proposed patches floating around on the mailing list have either already applied. One of them was applied upstream, then churned a bit and then reverted. If anybody could link me to some other patches or commits, I could give them a go.

I was taking a cursory glance at what's going into the upcoming 4.19.31 and there is one substantial ext4 change targeting inode stuff. Another ext4 change in there as well but seems to be related to "block" stuff. Not optimistic but but hopefully it has some benefit.
From my tests I can see that it is not per se ext4 issue but memory management of slab objects storing caches. All memory caches are affected I think, just ones related to ext4 operations are the easiest to fill in.

Having said that and anticipating that it might take some time to fix it wouldn't make sense to carry on with 4.14? The latest Pi release was 4.14.98 and it is already on 4.14.107
I'm certainly not smart enough to even understand the issue deeply or technically. I just troll through the RCs as they come to see what's coming down the pipe. From what I've read in the threads I posted, it *almost* sounds like a single component fix won't be the end all be all to fix the bug. I almost wonder if it'll be a combo fix of ext4 itself in tandem with the slab cache stuff. They are quite interwoven and interrelated that I would think significant changes in one would need some significant changes in the other. It's still interesting to see the gears turning in people heads on how to approach a fix. 50 different people will give you 50 different answers. :D

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5341
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Moving Linux kernel to 4.19

Tue Mar 26, 2019 2:22 pm

Potential workaround for the inode slab leak is now available in rpi-update kernel.
Affected users, please update and report back.

The workaround disables memory and IO cgroups which appears to be the cause of the leaks.

Return to “Advanced users”