Connecting the RPi to mobile network....

[Bosse_B]

I have an application I am working on where an RPi will be used to manage data collection in remote locations.
The system will run off of solar panel charged batteries and I have a few system design questions:

The RPi will manage the data collection and it will thus control some measuring equipment to measure at pre-configured times.
Configuration will be accomplished through the use of a web interface on the RPi.
The data collected needs to be transferred to an external server via the Internet and here where I need some advice:

Mobile data connection
1) Using a stand-alone mobile data router
I can include a mobile data router unit, which can connect to the mobile 4G data network, so the RPi can be connected via Ethernet on the private side and service personnel can connect to its WiFi when on location to directly interact with the system.
Or else the router may port forward the https port (443) to the Rpi which means that the system will be available not only if you are in close proximity but from everywhere.

2) Using a mobile data add-on for the RPi
Another more compact solution could be to use one of the mobile data interfaces that plugs into the GPIO pin header on the RPi itself.
I assume that this will put the RPi directly on the Internet and then I see a problem:
Where is the firewall?
Can the RPi be configured to act as a firewall too so no illegal activity can enter the RPi?

Solution 1 is more straight forward but has the drawback that it uses another box and quite possibly more power too.
Solution 2 seems more streamlined because it uses just as much power as is needed and no extra separate unit.
But what about security (fire-walling)?

Are there examples or discussions already on the subject available or do you have some advice?

[epoch1970]

I don't know about the add-ons themselves, but of course it is possible to use a Pi itself as a firewall.
It is a linux machine, like the extra router/firewall you might buy...

Besides simple firewall rules it is possible to segregate network interfaces and processes, with containers or more simply netns. Wireguard has a good write-up on this design.
As always with the Pi (or any machine in the wild), physical protection is paramount.

Design and testing might take a good deal of time, so it could be more cost effective in the end to rely on a "trusted" 1/3 party router.

[drgeoff]

Or else the router may port forward the https port (443) to the Rpi which means that the system will be available not only if you are in close proximity but from everywhere.

Be aware that some mobile operators use Carrier Grade NAT. That makes it impossible to directly call in. (To the RPi or any other data equipment.)

[Bosse_B]

Or else the router may port forward the https port (443) to the Rpi which means that the system will be available not only if you are in close proximity but from everywhere.

Be aware that some mobile operators use Carrier Grade NAT. That makes it impossible to directly call in. (To the RPi or any other data equipment.)

Thanks, I was unaware of this problem.
I am using a mobile broadband router at my summer home and I have set up Dynamic DNS for it, which works just fine to reach my router and the internal server from my home. In the summer home I have an RPi3 acting as an OpenVPN server as well as a few other monitoring devices, which are all reachable via the VPN. So I did not realize there was a potential problem.

But I guess that one would be hard pressed to issue all mobile broadband units a unique IPv4 address (not enough available)....

So how can one solve the connectivity problem then if that is in fact happening on the proposed data collection system?

[epoch1970]

Instead of port forwarding, have the Pi call home and connect to a known and available server.
Push data to it via HTTPS or setup a VPN server on the server and a client in the Pi.

[Bosse_B]

Yes, that is in the plan too. Measured data will be sent to a webserver using curl.
But configuration and control of the system needs a channel in the other direction...
Otherwise I have to implement some kind of system where the system "calls home" regularly and checks a flag on the webserver telling it that new config data are present, and if so downloads and uses these.
So in that case the system also is the active networking part.
It will introduce delays, though...

[Heater]

If your Pi are on a 3G/4G mobile connection they cannot be found from the internet. Often this is true for other connections from your ISP.

Ideally all devices would get an IPv6 address instead of Ipv4 and they would be public and we would have a proper peer-to-peer internet as it was intended rather than the lopsided arrangement of "clients" and "servers" we have today.

This is what IoT is all about. This situation is exploited such that IoT vendors can sell you devices which connect back to their cloud servers thus locking you into their service.

Solutions:

Get yourself a server on Digital Ocean, AWS, Google cloud, Azure, whatever. Then your Pi or other devices can connect out to that. Tha connection can be as an HTTP client to a web server, or use MQTT, or do whatever you like over a VPN like wireguard.

There need not be excessive delays, once a connection is established over HTTPS (web sockets), MQTT, VPN or whatever the server can push data to the device at any time with minimal delay, the connection is open all the time.

You could use the services of dataplicity https://www.dataplicity.com/devices/ to get yourself a connection to your command shell over the net. They will also tunnel to any web server on your device and make it accessible from the public internet.

You could use the services of https://www.balena.io (Makers of the famous Etcher SD image writer) to deploy such Pi applications.

Currently I'm rolling out a system with remote Pi over 4G using Dataplicity and application communicating back to a Digital Ocean server using NATS https://nats.io/

One might have the good fortune to have a public and static IP address from ones ISP at home. In which case at a pinch one could run the required services at home.

[emilfr]

If you're using unlimited mobile broadband and have a server at your permanent home, you can setup a vpn between the two, or use reverse ssh tunnel from your summer home to your permanent home.

[Bosse_B]

I decided to get a mobile phone cash card and use that in a USB mobile data modem for the RPi.
Then I will not go the route described above but rather have a config data store on a web-server which the RPi will check for config updates when it runs the cron job.
So I will have the config on the web and it will prepare the configurations such that when the cron job on the RPi starts the first task will be to check if there is any new config.
This way I will get less control but a simpler implementation.

[Bosse_B]

FOLLOW-UP
I have an additional question regarding connectivity:

How to set up the WiFi AP mode on RPi3/RPi4?
I would like to connect to the RPi using WiFi locally for configuration purposes.
So consider that it has been set up with a Mobile 4G broadband modem but nothing connected to the Ethernet port.
In this situation I would like to be able to use WiFi to connect to the RPi and access its Apache web-server where I will have configuration pages available.
But I don't want this AP to act as a bridge to the mobile network, just into the RPi itself.

The tutorials I have found to activate the WiFi AP function all are setting up the RPi to act as a router to the other network (normally eth0), but I just want to get WiFi running in order to configure its monitoring operations...

What is the easiest way to accomplish this?
Is it as simple as using this guide but only using steps 1-5 and stopping at step #6?

Or this guide and stop before section "Using the Raspberry Pi as an access point to share an internet connection (bridge)"?

Notice that this is a headless RPi, the only access to it is via SSH on eth0 (before deployment).
When in place it will only be accessed via WiFi...

[dickon]

What is the easiest way to accomplish this?
Is it as simple as using this guide but only using steps 1-5 and stopping at step #6?

I'd do that. Don't forget to remove the 'bridge=br0' line from hostapd.conf -- you won't be using it, after all -- and you'd best check what he means by

Code: Select all

denyinterfaces eth0
denyinterfaces wlan0

in dhcpcd.conf.

[Bosse_B]

I tried both tutorials but failed....
This is a summary what I did and what it resulted in:

Code: Select all

 sudo apt install dnsmasq hostapd
 sudo systemctl stop hostapd
 sudo systemctl stop dnsmasq
 sudo nano /etc/dhcpcd.conf

Add to end of file:

interface wlan0
    static ip_address=192.168.197.1/24
    nohook wpa_supplicant

Then:

Code: Select all

 sudo mv /etc/dnsmasq.conf /etc/dnsmasq.conf.orig
 sudo nano /etc/dnsmasq.conf

Add to new file:

interface=wlan0 # Use the require wireless interface - usually wlan0
dhcp-range=192.168.197.10,192.168.197.30,255.255.255.0,24h

Then:

Code: Select all

 sudo nano /etc/hostapd/hostapd.conf

Add to new file

interface=wlan0
driver=nl80211
ssid=NameOfNetwork
wpa_passphrase=PasswordOfNetwork
hw_mode=g
channel=7
wmm_enabled=0
macaddr_acl=0
auth_algs=1
ignore_broadcast_ssid=0
wpa=2
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP
rsn_pairwise=CCMP

Then

Code: Select all

 sudo nano /etc/default/hostapd

Modify daemon line:

DAEMON_CONF="/etc/hostapd/hostapd.conf"

Then

Code: Select all

 sudo service dhcpcd restart

No WiFi AP shows up...

Code: Select all

 sudo systemctl unmask host
 sudo systemctl unmask hostapd
 sudo systemctl enable hostapd
 sudo systemctl start hostapd

Result: No WiFi showing up
Also tried reboot without result.
Try to check what is going on:

Code: Select all

sudo systemctl status hostapd
● hostapd.service - Advanced IEEE 802.11 AP and IEEE 802.1X/WPA/WPA2/EAP Authenticator
   Loaded: loaded (/lib/systemd/system/hostapd.service; enabled; vendor preset: enabled)
   Active: activating (start) since Sun 2019-10-13 00:35:12 CEST; 234ms ago
Cntrl PID: 29928 (hostapd)
    Tasks: 1 (limit: 4915)
   Memory: 488.0K
   CGroup: /system.slice/hostapd.service
           └─29928 /usr/sbin/hostapd -B -P /run/hostapd.pid -B /etc/hostapd/hostapd.conf /etc/hostapd/hostapd.conf

Oct 13 00:35:12 rpi4-test hostapd[29928]: wlan0: AP-ENABLED
Oct 13 00:35:12 rpi4-test hostapd[29928]: Could not set channel for kernel driver
Oct 13 00:35:12 rpi4-test hostapd[29928]: Interface initialization failed
Oct 13 00:35:12 rpi4-test hostapd[29928]: wlan0: interface state UNINITIALIZED->DISABLED
Oct 13 00:35:12 rpi4-test hostapd[29928]: wlan0: AP-DISABLED
Oct 13 00:35:12 rpi4-test hostapd[29928]: wlan0: Unable to setup interface.
Oct 13 00:35:12 rpi4-test hostapd[29928]: wlan0: interface state ENABLED->DISABLED
Oct 13 00:35:12 rpi4-test hostapd[29928]: wlan0: AP-DISABLED
Oct 13 00:35:12 rpi4-test hostapd[29928]: wlan0: CTRL-EVENT-TERMINATING
Oct 13 00:35:12 rpi4-test hostapd[29928]: nl80211: deinit ifname=wlan0 disabled_11b_rates=0

I tried to edit the hostapd.conf file and use a different channel than 7, but to no avail. Same thing happens.
What am I doing wrong here?
ifconfig shows this:

Code: Select all

 ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.3.9  netmask 255.255.255.0  broadcast 10.0.3.255
        inet6 fe80::fffa:e103:a4c9:1230  prefixlen 64  scopeid 0x20<link>
        ether dc:a6:32:05:6c:d9  txqueuelen 1000  (Ethernet)
        RX packets 579054  bytes 139657892 (133.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 32546  bytes 9315898 (8.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 30  bytes 4254 (4.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 30  bytes 4254 (4.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether dc:a6:32:05:6c:db  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 58756  bytes 6241938 (5.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Must be something simple overlooked by the tutorials....

UPDATE:
I used WiFi analyzer on my phone and discovered that the AP appears and then disappears again in a second or so...

[epoch1970]

This looks ok to me overall. Did you configure the "wifi country" on the machine?
You can do that from raspi-config.
If the country isn't configured rfkill blocks wlan0 on Raspbian.

[Bosse_B]

I tried raspi-config and found the WiFi setting in Localization/I4
But when I tried it it displayed an error message:
"Could not communicate with wpa_supplicant "

Furthermore:
When I use WiFi analyzer on my phone I see the AP come up at irregular intervals of about 20-50 s and stay on for 3-4 seconds!
Then it disappears until the next "timeout" has passed.

I tried to shut down wlan0 in order for it not to disrupt the WiFi using

Code: Select all

sudo ip link set wlan0 down

and it seemingly succeeded because wlan0 does no longer show up in ifconfig.
However the AP network still appears in WiFi Analyzer at channel 11 every 20-50 s....

EDIT:
Currently I get this status response:

Code: Select all

sudo systemctl status hostapd
● hostapd.service - Advanced IEEE 802.11 AP and IEEE 802.1X/WPA/WPA2/EAP Authenticator
   Loaded: loaded (/lib/systemd/system/hostapd.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Sun 2019-10-13 17:27:47 CEST; 1s ago
  Process: 1502 ExecStart=/usr/sbin/hostapd -B -P /run/hostapd.pid -B $DAEMON_OPTS ${DAEMON_CONF} (code=exited, status=1/FAILURE)

sudo systemctl status dnsmasq
● dnsmasq.service - dnsmasq - A lightweight DHCP and caching DNS server
   Loaded: loaded (/lib/systemd/system/dnsmasq.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2019-10-13 15:18:15 CEST; 2h 10min ago
  Process: 419 ExecStartPre=/usr/sbin/dnsmasq --test (code=exited, status=0/SUCCESS)
  Process: 443 ExecStart=/etc/init.d/dnsmasq systemd-exec (code=exited, status=0/SUCCESS)
  Process: 547 ExecStartPost=/etc/init.d/dnsmasq systemd-start-resolvconf (code=exited, status=0/SUCCESS)
 Main PID: 546 (dnsmasq)
    Tasks: 1 (limit: 4915)
   Memory: 2.5M
   CGroup: /system.slice/dnsmasq.service
           └─546 /usr/sbin/dnsmasq -x /run/dnsmasq/dnsmasq.pid -u dnsmasq -r /run/dnsmasq/resolv.conf -7 /etc/dnsmasq.d,.dpkg-dist

Oct 13 15:18:15 rpi4-test dnsmasq[546]: warning: interface wlan0 does not currently exist
Oct 13 15:18:15 rpi4-test dnsmasq-dhcp[546]: DHCP, IP range 192.168.197.11 -- 192.168.197.30, lease time 1d
Oct 13 15:18:15 rpi4-test dnsmasq[546]: read /etc/hosts - 5 addresses
Oct 13 15:18:15 rpi4-test dnsmasq[546]: no servers found in /run/dnsmasq/resolv.conf, will retry
Oct 13 15:18:15 rpi4-test dnsmasq[547]: Too few arguments.
Oct 13 15:18:15 rpi4-test dnsmasq[547]: Too few arguments.
Oct 13 15:18:15 rpi4-test systemd[1]: Started dnsmasq - A lightweight DHCP and caching DNS server.
Oct 13 15:18:35 rpi4-test dnsmasq[546]: reading /run/dnsmasq/resolv.conf
Oct 13 15:18:36 rpi4-test dnsmasq[546]: using nameserver 208.91.112.53#53
Oct 13 15:18:36 rpi4-test dnsmasq[546]: using nameserver 208.91.112.52#53

Really confusing...

[maverik0106]

Hi,

I was in process of doing this exact thing a while ago, setting up my raspberry pi 3+ as an access point and router as well as using a hologram.io 3g/4g dongle to have this outside of my local network for data gathering.

I got the first part, but like you I haven't been able to properly connect to the raspberry pi through a tunnel... I tried openvpn but it never works, I need A LOT more understanding on networks to set up the openvpn properly...

Anyway long story short, I can give you the link so you that you can set up your rpi as an access point and allowing routing on both wifi and ethernet interfaces. It uses systemd which apparently it's the new big thing, also, posts by this guy are pretty awesome on networking so hope this helps

Link

I'll keep an eye in case you get your second part working... since I'd be interested to see what your approach was/is... will continue to try different setups for mine too

[Bosse_B]

Anyway long story short, I can give you the link so you that you can set up your rpi as an access point and allowing routing on both wifi and ethernet interfaces. It uses systemd which apparently it's the new big thing, also, posts by this guy are pretty awesome on networking so hope this helps

Thanks, I will have a look.
But my use case is a bit different:
I want to enable the RPi to expose a WiFi AP that is NOT routed out and only allows access to the RPi itself.
I need that for configuration purposes when in the vicinity of the RPi.
It is a bit like how ChromeCast does it when first powered up and it creates this hotspot which Google Home connects to in order to configure its wifi connectivity to the Internet.
In my case I want the AP on the RPi to allow connection to the config website I will run on the RPi itself so it can be set up from any phone or tablet nearby (with the proper login of course).
I also would like to be able to enable the AP with a physical switch connected to some GPIO pin on the RPi...

Will get back to your link in the afternoon/evening (now in Texas so it is mid day here).

Concerning OpenVPN I have set it up as a server on many RPi units over the years and it works just fine.
RPi is a useful OpenVPN platform!
But I have never used OpenVpn client functions on a Linux system ever, so I cannot say how that would work.

[Bosse_B]

I have now reverted what I did before so I am back at a "clean" state.
Had a look at your stack-exchange link and found a further link there to this description where setting up several different modes are shown.
This is a massive webpage since it covers a lot of configs.
Due to its massiveness I have copied the relevant parts for my use case here.

Specifically there is a mode that mimicks the one I am after, namely using the WiFi AP as a config tool connecting to the RPi only (no further routing at all).
Basically this seems to entail the following:
First General Setup that needs always to be done:

General Setup
Switch over to systemd-networkd
For detailed information look at [urlhttps://raspberrypi.stackexchange.com/a/78788/79866](1)[/url]. Here only in short. Execute these commands:

Code: Select all

# disable classic networking
rpi ~$ sudo -Es
rpi ~# systemctl mask networking.service dhcpcd.service
rpi ~# mv /etc/network/interfaces /etc/network/interfaces~
rpi ~# sed -i '1i resolvconf=NO' /etc/resolvconf.conf

# enable systemd-networkd
rpi ~# systemctl enable systemd-networkd.service systemd-resolved.service
rpi ~# ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

Configure wpa_supplicant as access point

To configure wpa_supplicant as access point create this file with your settings for country=, ssid=, psk= and maybe frequency=. You can just copy and paste this in one block to your command line beginning with cat and including both EOF (delimiter EOF will not get part of the file):

Code: Select all

rpi ~# cat > /etc/wpa_supplicant/wpa_supplicant-wlan0.conf <<EOF
country=DE
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

network={
    ssid="RPiNet"
    mode=2
    frequency=2437
    #key_mgmt=NONE   # uncomment this for an open hotspot
    # delete next 3 lines if key_mgmt=NONE
    key_mgmt=WPA-PSK
    proto=RSN WPA 
    psk="password"
}
EOF

rpi ~# chmod 600 /etc/wpa_supplicant/wpa_supplicant-wlan0.conf
rpi ~# systemctl disable wpa_supplicant.service
rpi ~# systemctl enable wpa_supplicant@wlan0.service

Finished General setup. Go back.

Then the setup for my specific use (other cases in the quoted webpage):

Setting up a stand alone access point

Example for this setup:

Code: Select all

                 wifi
mobile-phone <~.~.~.~.~> (wlan0)RPi(eth0)
            \             /
           (dhcp)   192.168.4.1

Do "General setup" then create the following file to configure wlan0.
We only have the access point. There is no ethernet device configured.

Code: Select all

rpi ~# cat > /etc/systemd/network/08-wlan0.network <<EOF
[Match]
Name=wlan0
[Network]
Address=192.168.4.1/24
DHCPServer=yes
EOF

If you want this then reboot.
That's it.

What I do not find here is how to set up the range of IP addresses connected devices will get...
It seems like the IP mentioned in the config is what the RPi gets, but what is served out to the connected device?
Any ideas?

[epoch1970]

using the WiFi AP as a config tool connecting to the RPi only (no further routing at all).

Maybe it is time to regroup.
From a clean Raspbian install, apply this: https://www.raspberrypi.org/documentati ... s-point.md reading from the top of the page up to title "Add routing and masquerade". Disregard anything below that title.
That gives you an isolated AP creating a network for wifi clients. Clients will not have internet access.

[Bosse_B]

using the WiFi AP as a config tool connecting to the RPi only (no further routing at all).

Maybe it is time to regroup.
From a clean Raspbian install, apply this

Well, as described above I actually have used this tutorial and it got me to an unstable situation.
The problem now is how can I revert to the "clean Raspbian install" state?
I have tried to revert in the following way:

Code: Select all

sudo reboot
sudo systemctl stop hostapd
sudo systemctl stop dnsmasq
sudo apt remove dnsmasq hostapd
sudo apt autoremove
sudo reboot
cd /etc/wpa_supplicant/
sudo mv wpa_supplicant-wlan0.conf wpa_supplicant-wlan0.conf.bak
sudo cp wpa_supplicant-wlan0.conf.bak wpa_supplicant-wlan0.conf
sudo nano /etc/wpa_supplicant/wpa_supplicant-wlan0.conf

Remove old stuff only leaving this:

Code: Select all

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=US

And in wpa_supplicant.conf I only have this:

Code: Select all

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=SE

network={
        ssid="*******"
        psk="********"
}

Need to head out now....

[maverik0106]

The only problem with modifying wpa_supplicant while using the setup from the link I gave you, is that systemd may override some if not most of the information on the wpa_supplicant, since it creates its own supplicant_wlan0 or whichever interface you decided to set up. I know it’s from a completely different hardware, but I was also trying to find something like that a couple months ago for an Arduino, and I came across something called a wifi manager Link here
Maybe if you’re experienced enough you might be able to translate it into Raspberry code

Forgot to mention, there is a way to give it an IP range if you’re using the systemd way:

Code: Select all

rpi ~# cat > /etc/dnsmasq.conf <<EOF
interface=br0
  dhcp-range=192.168.50.128,192.168.50.164,255.255.255.0,24h
  dhcp-option=option:router,192.168.50.1
  dhcp-option=option:dns-server,8.8.8.8,1.1.1.1
EOF

[Bosse_B]

I do have experience with the ESP8266 modules, in fact I have built quite a number of projects using these IoT wonders and Arduino environment.
But I have coded my own functions to start up in connected state or as a WiFi AP for configuration purposes. Not using WiFiManager since it is not obvious how to get out of the automated functions.
But even so I have used top level functions in available libraries so there is not much that I can translate into Raspbian...

Concerning my broken system I have had to start over with a fresh Raspbian image SDcard to get it sorted.
I could use a USB caddy for my old SDcard and mount it to the RPi4 and then I could transfer the /home/pi content.
I also made a history dump to file before shutting down the old system so I could use this on the new system to figure out which apt installs I had done and in which order to build my previous system. This way I could duplicate the apt installations on the new system (except the last few ones which broke the WiFi functions).

So now I am back in business with my main development RPi.
I also created a lite version in the process (no GUI), which I will use in order to try and figure out what needs to be done in order to:
1) Create a WiFi access point on the Pi.
2) Connect to mobile broadband using a USB modem
All without risking my development system...