schwapp
Posts: 8
Joined: Mon Oct 05, 2015 1:23 pm

Internet traffic monitor

Tue Oct 06, 2015 8:07 am

I'm searching for a solution for the network in our youth club, where several members get internet access on our WLAN. The MAC addresses of their WLAN interfaces are registered and mapped to their IP addresses, individual IP addresses of the members are fixed, so a mapping from IP address to user is available (we do not need individual DHCP functionality, it is not a classical "free WLAN", where anybody can log in).

What we want: In case illegal activities from this network are notified by a provider or a lawyer is sueing us for this misdoing (this can happen at any time unfortunately...) we have to be able to find out who of the members has accessed which external IP address sent off from the internal WLAN. Therefore we need a 24/7 logging of the IP accesses of this network, say which internal IP is accessing which external IP and at which time.

I would like to use a Raspberry PI for that because I think it is easy to connect to the router, can be powered by the USB connector of the router.
My technical background: I've already installed and running a Linux server (Ubuntu) at home, so I'm not a complete newbie. But I've neither used RPI before nor found any solution for such a network monitor. All of the solutions I've found til now by Google is different from my use case (e.g. checking if the internet connection is still alive, check current internet speed, checking current data throughput, ...).

So does anybody have an idea how I can monitor the network activities with the raspberry pi?

User avatar
lmarmisa
Posts: 1233
Joined: Thu Feb 14, 2013 2:22 am
Location: Jávea, Spain

Re: Internet traffic monitor

Tue Oct 06, 2015 4:14 pm

First of all, you need a router with the capability for logging the information you want. I do not know wich is the model of your router. Cheap routers provided by ISP will probably not support such capabilities.

If your router is too basic, I recommend to install a Mikrotik router 951G-2HnD (cheap, powerfull, wifi and reliable):

http://routerboard.com/RB951G-2HnD

You could use your RPi for storing the remote syslog of your router:

https://aacable.wordpress.com/2011/11/2 ... og-server/

Mikrotik routers runs RouterOs, a very powerfull OS oriented to routers based on Linux. You will be able to define rules for logging the traffic monitor according your needs (http, https, ftp, etc) and store the information in your RPi.

http://www.mikrotik.com/pdf/what_is_routeros.pdf

kcx
Posts: 15
Joined: Tue Apr 21, 2015 10:29 pm
Location: Central Texas

Re: Internet traffic monitor

Tue Oct 06, 2015 4:53 pm

It sounds like what you want is to collect netflow from the router/pi and export that to a host for later potential analysis.

I haven't used it, but it seems that ipt-netflow will allow you to export flows. You would want to log this to a USB hd or off the pi as a SD card won't hold up very long to that many writes.

schwapp
Posts: 8
Joined: Mon Oct 05, 2015 1:23 pm

Re: Internet traffic monitor

Wed Oct 07, 2015 11:10 am

lmarmisa wrote:First of all, you need a router with the capability for logging the information you want. I do not know wich is the model of your router. Cheap routers provided by ISP will probably not support such capabilities.

If your router is too basic, I recommend to install a Mikrotik router 951G-2HnD (cheap, powerfull, wifi and reliable):

http://routerboard.com/RB951G-2HnD

You could use your RPi for storing the remote syslog of your router:

https://aacable.wordpress.com/2011/11/2 ... og-server/

Mikrotik routers runs RouterOs, a very powerfull OS oriented to routers based on Linux. You will be able to define rules for logging the traffic monitor according your needs (http, https, ftp, etc) and store the information in your RPi.

http://www.mikrotik.com/pdf/what_is_routeros.pdf
Unfortunately we've no influence to the used router, as it is determined by our internet provider. We have to use a Fritz!Box (AVM), which is in principle quite a good machine, but the firmware is locked by the internet provider. In general it is possible to trace the internet traffic metadata with the FritzBox, but this requires to log in to the web-interface of the router and download a log file from the router to the computer which uses this web interface currently. The logging starts when it is initiated by the web-interface, so a 24/7 logging is not possible.

Thanks for the response, but this is not possible for our use case.

fruitoftheloom
Posts: 21084
Joined: Tue Mar 25, 2014 12:40 pm
Location: Delightful Dorset

Re: Internet traffic monitor

Wed Oct 07, 2015 11:24 am

Last edited by fruitoftheloom on Wed Oct 07, 2015 11:30 am, edited 1 time in total.
Retired disgracefully.....
This at present is my daily "computer" https://www.asus.com/us/Chrome-Devices/Chromebit-CS10/

schwapp
Posts: 8
Joined: Mon Oct 05, 2015 1:23 pm

Re: Internet traffic monitor

Wed Oct 07, 2015 11:27 am

kcx wrote:It sounds like what you want is to collect netflow from the router/pi and export that to a host for later potential analysis.

I haven't used it, but it seems that ipt-netflow will allow you to export flows. You would want to log this to a USB hd or off the pi as a SD card won't hold up very long to that many writes.
Hm, interesting!
I'm a little scared because compilation seems to be necessary...
Can it be that I also need to set the DHCP Server task from the router to the Raspberry? Does the RPI have to route all traffic then? Maybe this is too much load for the RPI?

User avatar
lmarmisa
Posts: 1233
Joined: Thu Feb 14, 2013 2:22 am
Location: Jávea, Spain

Re: Internet traffic monitor

Wed Oct 07, 2015 11:37 am

Do you have access to the router menu (normally by http)?. Can you disable the wifi of your router?.

In such case you could develop your own wifi access point/router in your RPi (or add a new Mikrotik wifi router) and store the logs according your needs. This is the solution proposed by fruitoftheloom but it will require some development. The project using a second router (Mikrotik) will be not complex.

Another option would be to substitute the current router by a new one, yours. You will need to know the config parameters for the access to the Internet. What kind of connection have you?. ADSL?
Last edited by lmarmisa on Wed Oct 07, 2015 11:56 am, edited 2 times in total.

schwapp
Posts: 8
Joined: Mon Oct 05, 2015 1:23 pm

Re: Internet traffic monitor

Wed Oct 07, 2015 11:40 am

Cool, this may be an interesting alternative. But how can I get the routing information out of it? The tutorial does not say anything about it. Do I have to combine it with ipt netflow?

schwapp
Posts: 8
Joined: Mon Oct 05, 2015 1:23 pm

Re: Internet traffic monitor

Wed Oct 07, 2015 11:47 am

lmarmisa wrote:Do you have access to the router menu (normally by http)?. Can you disable the wifi of your router?.

In such case you could develop your own wifi access point/router in your RPi and store the logs according your needs. This is the solution proposed by fruitoftheloom but it will require some development.

Another option would be to substitute the current router by a new one, yours. You will need to know the config parameters for the access to the Internet. What kind of connection have you?. ADSL?
Yes, I can (and will) disable the WIFI, because the actual location where the WLAN shall work is ~100m away from the router, there is a LAN cable available and a remote WLAN access point has to do the job.

The thing is, I cannot find information in fruitoftheloom's link on how to get the traffic metadata information out of the RPI router...

Unfortunately it is (still) possible here in Germany that the internet provider does not reveal the access data and therefor use a certain mandatory router for our internet access.
The connection is a fiber glass connection, transformed to normal ADSL.

User avatar
lmarmisa
Posts: 1233
Joined: Thu Feb 14, 2013 2:22 am
Location: Jávea, Spain

Re: Internet traffic monitor

Wed Oct 07, 2015 12:06 pm

I recommend this solution:

1) Disable wifi in your current router.

2) Add a second router. This device should support powerfull log filters and remote log (rsyslog). This second router will be connected to the router Friz!Box by Ethernet. The router Mikrotik 951G-2HnD could be a candidate.

3) Collect the log information in your RPi.

This project is feasible but it still requires a small effort for configuring all your requeriments.

Internet <-> Router Friz!Box <-> Your router (logs are generated here) <-> Wifi Access Point (included in your router?) <-> Users

schwapp
Posts: 8
Joined: Mon Oct 05, 2015 1:23 pm

Re: Internet traffic monitor

Wed Oct 07, 2015 1:11 pm

lmarmisa wrote:I recommend this solution:

1) Disable wifi in your current router.

2) Add a second router. This device should support powerfull log filters and remote log (rsyslog). This second router will be connected to the router Friz!Box by Ethernet. The router Mikrotik 951G-2HnD could be a candidate.

3) Collect the log information in your RPi.

This project is feasible but it still requires a small effort for configuring all your requeriments.

Internet <-> Router Friz!Box <-> Your router (logs are generated here) <-> Wifi Access Point (included in your router?) <-> Users
You are right, maybe it is easier to buy a dedicated hardware for recording the traffic. In that case an additional Raspberry Pi just receives the logs sent by the 2nd router - so to say as a permanently running computer in the network, right?

User avatar
lmarmisa
Posts: 1233
Joined: Thu Feb 14, 2013 2:22 am
Location: Jávea, Spain

Re: Internet traffic monitor

Wed Oct 07, 2015 1:22 pm

schwapp wrote:
lmarmisa wrote:I recommend this solution:

1) Disable wifi in your current router.

2) Add a second router. This device should support powerfull log filters and remote log (rsyslog). This second router will be connected to the router Friz!Box by Ethernet. The router Mikrotik 951G-2HnD could be a candidate.

3) Collect the log information in your RPi.

This project is feasible but it still requires a small effort for configuring all your requeriments.

Internet <-> Router Friz!Box <-> Your router (logs are generated here) <-> Wifi Access Point (included in your router?) <-> Users
You are right, maybe it is easier to buy a dedicated hardware for recording the traffic. In that case an additional Raspberry Pi becomes unnecessary, right?
The Rpi could be still neccesary for storing the traffic recorded by your router.

If you select a router for recording traffic with a small storage capability, you will probably need a RPi or other similar computer.

jkw
Posts: 12
Joined: Thu Oct 02, 2014 5:20 pm

Re: Internet traffic monitor

Wed Oct 07, 2015 6:51 pm

Just a stupid idea, but have you thought about some kind of tshark running on a rpi? Just put the wireless interface in listen mode an record all the traffic. You should maybe limit it to source and destination IP but that should give you all you need, without extra hardware or have I skipped an important limitation here?
JKW

schwapp
Posts: 8
Joined: Mon Oct 05, 2015 1:23 pm

Re: Internet traffic monitor

Sun Nov 08, 2015 6:25 pm

Tanks for all your hints, finally I've found a quite fitting solution for my use case - including the Raspberry PI!
The used router is - as I've already mentioned - a FritzBox. This is a quite powerful device also, and there is a quite good support community for that. I also wrote already, that the FritzBox is in general able to trace the entire traffic into a Wireshark file, but this is only possible via the Web-Interface.

But: There are resourceful people out there, who know how to tweak this device and know alternative - not officially documented - means to access features of the router.

The keyword for that is "fritzcap", means capture for the fritzbox. If you google for it, you find the native implementation here (mostly in German) and - what is very interessting for me - the python porting hosted by Google. This port is not so powerful, at least according to the documentation, but my use case is a very simple one and fits perfectly to the Raspberry Pi, as python is a standard tool for it and also easy to modify.

Fritzcap is running on a computer within the network, we can use a Raspberry for it which is directly connected to the LAN-Port of the FritzBox. The script accesses the FritzBox just with a special URL which lead the FritzBox to respond with the wanted trace data. This data is simply captured by the script and stored to the local Raspberry Pi user filesystem.

As the original python script does more than needed for my task and needs a user interaction (press key button) and a filing which is not suitable for me, I've modified the script accordingly. You can find my modified version below.
If you want to use it, please download all python files from the mentioned Google-Server above, because the script still needs the external libraries provided with fritzcap.

My traffic capture is now running 24/7, using my first Raspberry Pi!!! Great!

Code: Select all

#!/usr/bin/python
# -*- coding: iso-8859-1 -*-
################################################################################# 
# Simple FritzCap python port
# Simplifies generation and examination of traces taken from AVM FritzBox and/or SpeedPort
# Traces can be examined using WireShark
# (c) neil.young 2010 (spongebob.squarepants in http://www.ip-phone-forum.de/)
# based on the Windows GUI exe with same name
##################################################################################
# Copyright (c) 2010, neil.young
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#  * Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
#  * Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#  * Neither the name of the <organization> nor the
#    names of its contributors may be used to endorse or promote products
#    derived from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY
# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
##################################################################################

import urllib, re, hashlib, sys, datetime, os, locale, time

script_dir = os.path.dirname(os.path.realpath(__file__)) 
sys.path.append(script_dir + '/core')

from tracer import Tracer
from pcap_parse import PcapParser
from g711_decoder import G711Decoder

# Configuration (just change here) ###############################################
boxname         = 'xxx.xxx.xx.xx'       # ip of the router  
password        = 'yyyyyyyyyyyyyyyyy'   # your password, adapt
protocol        = 'https'               # or http
capfolder       = 'captures'            # plus subfolders according to day, month, year, hour, minute
pre_05_50       = False                 # for versions < FRITZ!OS 05.50
capture_seconds = 60*60                 # capture data for x seconds each
locale.setlocale(locale.LC_ALL,'de_DE.UTF-8')  # set your locale
##################################################################################

# Commands
default_login   = 'getpage=../html/de/menus/menu2.html&errorpage=../html/index.html&var:lang=de&var:pagename=home&var:menu=home&=&login:command/password=%s'

if pre_05_50:
  sid_challenge   = protocol + '://' + boxname + '/cgi-bin/webcm?getpage=../html/login_sid.xml'
  sid_login_url   = protocol + '://' + boxname + '/cgi-bin/webcm'
  sid_login_parm  = 'login:command/response=%s&getpage=../html/login_sid.xml'
else:
  sid_challenge   = protocol + '://' + boxname + '/login_sid.lua'
  sid_login_url   = sid_challenge
  sid_login_parm  = 'response=%s&page=&username='
  
start = '?start=1&start1=Start'
stop  = '?stop=1&stop1=Stop'

# Main
def main():

  global capfile
  SID = ''                  # Required later         
  
  while True:               # Run endless
    
    print " "

    try:
      # Try to get a session id SID
      sid = urllib.urlopen(sid_challenge)
      if sid.getcode() == 200: 
             
        # Read and parse the response in order to get the challenge (not a full blown xml parser)
        challenge = re.search('<Challenge>(.*?)</Challenge>', sid.read()).group(1)

        # Create a UTF-16LE string from challenge + '-' + password, non ISO-8859-1 characters will except here (e.g. EUR)
        challenge_bf = (challenge + '-' + password).decode('iso-8859-1').encode('utf-16le')
        
        # Calculate the MD5 hash
        m = hashlib.md5()
        m.update(challenge_bf)
    
        # Make a byte response string from challenge + '-' + md5_hex_value 
        response_bf = challenge + '-' + m.hexdigest().lower()
        
        # Answer the challenge
        login = urllib.urlopen(sid_login_url, sid_login_parm % response_bf)
        
        if login.getcode() == 200:
          SID = re.search('<SID>(.*?)</SID>', login.read()).group(1);
          print "Login OK, SID=%s" % SID
        else:
          print "Could not login"
          return
        
      else:
        print 'Wrong SID code...'

      # end of if sid.getcode() == 200

    except:
      print 'Legacy login...'
      # Legacy login
      command = urllib.urlopen(protocol + '://' + boxname + '/cgi-bin/webcm', default_login % password)
      response = command.read()
      # Right now I don't know how to check the result of a login operation. So I just search for the errorMessage
      if command.getcode() == 200:
        try:
          result = urllib.unquote(re.search('<p class="errorMessage">(.*?)</p>', response).group(1).decode('iso-8859-1')).replace("&nbsp;"," ")
        except:
          result = ''
        print 'Login attempt was made. %s' % result
     
    # end of try/except:
   
    # Create capfile folders
    folder = script_dir + '/' + capfolder + '/' + datetime.datetime.now().strftime('%Y') + '/' + datetime.datetime.now().strftime('%m_%B') + '/' + datetime.datetime.now().strftime('%d_%A')

    if not os.path.exists(folder):
      os.makedirs(folder)
    capfile = folder + '/capture_' + (datetime.datetime.now().strftime('%H%M')) + 'Uhr.eth'

    # Start tracer thread, wait for given number of seconds...
    if SID != '':
      Tracer(protocol + '://' + boxname + '/cgi-bin/capture_notimeout' + start + "&sid=%s" % SID, capfile).start()
    else:
      Tracer(protocol + '://' + boxname + '/cgi-bin/capture_notimeout' + start, capfile).start()
      
    print 'Trace started at %s o\'clock, capture for %d minutes...' % ((datetime.datetime.now().strftime('%H:%M')), capture_seconds/60)

    time.sleep(capture_seconds)

    # Clean stop
    print 'Stopping trace'
    if SID != '':
      urllib.urlopen(protocol + '://' + boxname + '/cgi-bin/capture_notimeout' + stop + "&sid=%s" % SID)
    else:
      urllib.urlopen(protocol + '://' + boxname + '/cgi-bin/capture_notimeout' + stop)
    print 'Capture done'

  # end of while True

  print 'All done'
      
if __name__ == '__main__': 
  main()  
As the script is running in an endless loop, I've added some small bash scripts for the Raspberry Pi which can be used for manual start and stop of the script and a restart script for usage e.g. in a cronjob. These bash-scripts need to reside in the same directory as the python script.

Start the script:

Code: Select all

#!/bin/bash

python $(dirname $0)/fritzcap.py &
Stop the script:

Code: Select all

#!/bin/bash

process_output=$(ps aux | grep "fritzcap.*\.py" | grep -v grep)
if [ -n "$process_output" ]; then

    echo
    echo 'This is the fritzcap process:'
    echo $process_output

    process_pid=$(echo $process_output|awk '{print $2}') 

    echo "Kill the fritzcap-script with process id $process_pid..."
    kill $process_pid

else
    echo 'Process fritzcap.py not found for killing.'
fi
Restart the script:

Code: Select all

#!/bin/bash

$(dirname $0)/stop_fritzcap.sh

echo
echo Wait for 10 seconds before restarting fritzcap...
sleep 10

$(dirname $0)/start_fritzcap.sh
Have fun!

Chris14
Posts: 3
Joined: Wed Dec 02, 2015 1:14 pm

Re: Internet traffic monitor

Thu Dec 03, 2015 1:12 pm

schwapp wrote:Tanks for all your hints, finally I've found a quite fitting solution for my use case - including the Raspberry PI!
The used router is - as I've already mentioned - a FritzBox.
How did you manage the login Username?

I tried everything as described but get an error

Code: Select all

pi@raspberrypi ~ $ Login OK, SID 0000000000000000
Trace started, abandon with <ENTER>
Traceback (most recent call last):
  File "./fritzcap.py", line 160, in <module>
    main()
  File "./fritzcap.py", line 143, in main
    raw_input()
EOFError: EOF when reading a line
Trace finished server side
^C
pi@raspberrypi ~ $
This is especially irritating, since its the same with or without pwd.

schwapp
Posts: 8
Joined: Mon Oct 05, 2015 1:23 pm

Re: Internet traffic monitor

Thu Dec 03, 2015 1:32 pm

How did you manage the login Username?
I tried everything as described but get an error
This is especially irritating, since its the same with or without pwd.
I don't think that login with username is supported. You have to switch of login with username in the fritzbox and just the password. The SID 0000000000000000 already indicates that the start does not work.

Return to “Networking and servers”