AeroScan - Wireless Network Q&P Monitoring

Overview

AeroScan is an IoT-based system designed to monitor the Quality and Performance (Q&P) of wireless network infrastructure. This project utilizes Raspberry Pi devices deployed across a site to collect real-time network metrics. These metrics are then sent to a central server for aggregation by Prometheus and visualization via Grafana dashboards, providing valuable insights into network health, helping to identify potential issues, and ensuring a reliable wireless experience.

The system features a central registration service (pi-registrar) for dynamic discovery of Raspberry Pi nodes by Prometheus. Configuration of the Raspberry Pis, including sensitive details like WiFi credentials and API keys, can be managed remotely using Ansible from the central server.

Features

• Distributed Monitoring: Uses Raspberry Pi devices as network sensors.
• Centralized Data Collection & Visualization: Employs Prometheus for metrics scraping/storage and Grafana for dashboards.
• Dynamic Pi Discovery: A custom pi-registrar service allows Prometheus to dynamically find active Raspberry Pi nodes.
• Secure Communication:
  • Pis report to the registrar over HTTPS.
  • Registrar uses an API key for authenticating Pi registration.
  • Prometheus scrapes the registrar over HTTPS.
• Comprehensive Metrics Collection:
  • Network Latency (Ping RTT, TTL) & Jitter.
  • Internet Bandwidth (Upload/Download via Speedtest).
  • Connected WiFi AP Signal Strength & Link Quality (via iwconfig).
  • Nearby Wireless Access Points Scan (SSID, BSSID, Channel, Signal Strength via nmcli).
  • Device IP Addresses (for specified wireless and LAN interfaces).
• Remote Configuration (via Ansible):
  • Ability to update Pi configurations (WiFi credentials, API keys, script settings) from the central server using Ansible.
  • Dynamic Ansible inventory sourced from the pi-registrar service.
• Unique Device Identification: Each Raspberry Pi sensor uses a persistent unique identifier.
• GPIO Interaction (Optional): Functionality to reset the device identifier using physical buttons.

Technology Stack

• Sensor Hardware (Raspberry Pi):
  • Raspberry Pi (e.g., Zero WH, 3B+, 4B)
  • MicroSD Card (16GB+ recommended)
  • Power Supply
  • (Optional) Push Buttons and wiring for GPIO.
• Sensor Operating System:
  • Custom Raspberry Pi OS image (Debian-based).
  • Assumes NetworkManager is installed and managing network interfaces.
• Sensor Software (Raspberry Pi - pi/software/main.py):
  • Python 3
  • prometheus_client (for exposing metrics)
  • speedtest-cli (for bandwidth tests)
  • RPi.GPIO (for button interaction)
  • requests (for reporting to registrar)
• Sensor System Tools (on Pi):
  • ping
  • ip (from iproute2 package)
  • iwconfig (from wireless-tools package for connected AP metrics)
  • nmcli (NetworkManager CLI for WiFi scanning and management)
  • git (for pulling script updates)
• Central Server (Ubuntu recommended):
  • Docker & Docker Compose
  • Services (Managed by Docker Compose):
    • Prometheus (Scraping, Storage)
    • Grafana (Visualization)
    • Pi-Registrar (Custom Flask API for Pi registration and service discovery)
  • Ansible (For remote configuration of Pis, runs on the Docker host or in a container)
• Version Control: Git / GitHub

Architecture Overview

1. Raspberry Pi Node (main.py):
   • Pulls the latest version of AeroScan/pi/software/main.py from Git on boot.
   • Reads local configuration files for registrar URL, API key, and RPi Connect device name.
   • Periodically collects network metrics using system tools (ping, speedtest-cli, iwconfig, nmcli).
   • Exposes these metrics via a Prometheus exporter endpoint (e.g., http://<pi_ip>:8000/metrics).
   • Registers itself (identifier, IP, metrics port) with the central pi-registrar service over HTTPS using an API key.
2. Central Server:
   • Pi-Registrar Service (Docker):
     • A Flask API listens for registrations from Pi nodes.
     • Exposes an /targets endpoint for Prometheus HTTP Service Discovery (providing a list of active Pis).
     • Exposes an /ansible_inventory endpoint (dynamic inventory source for Ansible).
     • Serves traffic over HTTPS using self-signed certificates generated by setup_server.sh.
   • Prometheus (Docker):
     • Configured to use HTTP Service Discovery, querying the pi-registrar's /targets endpoint to find Pis to scrape.
     • Scrapes metrics from each registered Pi.
     • Stores time-series data.
   • Grafana (Docker):
     • Connects to Prometheus as a data source.
     • Provides dashboards for visualizing network metrics from all Pis.
   • Ansible (on Docker Host):
     • Uses a dynamic inventory script that queries the pi-registrar's /ansible_inventory endpoint.
     • Allows administrators to run playbooks to configure Pis (e.g., update WiFi, API keys, manage files).

Setup and Installation

This project involves setting up a Central Server and one or more Raspberry Pi nodes.

A. Central Server Setup (Ubuntu Recommended)

1. Prerequisites:

   • An Ubuntu server (or other Linux distribution capable of running Docker and Ansible).
   • Docker and Docker Compose installed.

     # Example for Ubuntu:
     sudo apt update
     sudo apt install -y docker.io docker-compose
     sudo systemctl start docker
     sudo systemctl enable docker
     sudo usermod -aG docker $USER # Add your user to docker group, then log out/in
     

   • Ansible installed (see "C. Ansible for Configuration Management" section below or install directly).
   • Git installed (sudo apt install -y git).
   • OpenSSL installed (usually present, for certificate generation).

2. Clone AeroScan Repository:

   git clone https://github.com/Aero-Scan/AeroScan.git
   cd AeroScan
   

3. Run Server Setup Script: This script generates SSL certificates for pi-registrar and an .env file with the API key for Pi registration.

   chmod +x setup_server.sh
   ./setup_server.sh
   

   • Follow the prompts (e.g., for Server IP to embed in the certificate's Subject Alternative Name).
   • Note the PI_REGISTER_API_KEY generated – you'll need this for the Raspberry Pis.
   • This will create/populate:
     • Docker/config/certs/ (pi-registrar's cert and key)
     • Docker/config/prometheus_certs/ (copy of pi-registrar's public cert for Prometheus)
     • ansible/config/ (copy of pi-registrar's public cert for Ansible inventory script)
     • Docker/.env (with PI_REGISTER_API_KEY)

4. Start Docker Services: Navigate to the Docker directory and start the services:

   cd Docker/
   docker-compose up -d --build
   

   • Verify services are running: docker-compose ps
   • Check logs if needed: docker-compose logs pi-registrar, docker-compose logs prometheus.

5. Configure Grafana:

   • Access Grafana at http://<your_server_ip>:3000 (default admin/admin, change password).
   • Add Prometheus as a data source:
     • URL: http://prometheus:9090 (uses Docker's internal DNS).
     • Click "Save & Test".
   • Create or import dashboards to visualize AeroScan metrics.

B. Raspberry Pi Node Setup

This assumes you have a custom Raspberry Pi OS image that:

• Is based on a Debian-derived system (like Raspberry Pi OS).
• Has networking configured to connect to your network (e.g., Ethernet or initial WiFi).
• Has git installed.
• On boot, automatically clones/pulls the AeroScan repository (e.g., into ~/AeroScan) and runs ~/AeroScan/pi/software/main.py.
• The user running main.py (e.g., pi or aeroscan) has necessary permissions.

1. Required Packages on the Raspberry Pi Image: Ensure these are installed (can be part of your image build process or installed manually once):

sudo apt update
sudo apt install -y python3 python3-pip git network-manager wireless-tools iproute2
# For main.py Python dependencies:
sudo pip3 install prometheus-client speedtest-cli RPi.GPIO requests urllib3

• NetworkManager is crucial if main.py uses nmcli for WiFi scanning and you plan to manage WiFi with Ansible via nmcli. Ensure it's the active network management service (this might involve disabling dhcpcd).

2. Configuration Files on Each Raspberry Pi: These files need to be created in /var/local/ on each Pi. This can be done manually for initial setup, or ideally, managed by Ansible.

• (Required) Registrar URL Configuration:

  • File: /var/local/network_monitor_registrar_url.txt
  • Content: The HTTPS URL of your pi-registrar service.

    https://<YOUR_CENTRAL_SERVER_IP_OR_DNS>:5001/register
    

    (Replace <YOUR_CENTRAL_SERVER_IP_OR_DNS> with the IP/hostname you configured in the certificate SAN during setup_server.sh).

• (Required) API Key for Registration:

  • File: /var/local/network_monitor_api_key.txt
  • Content: The exact PI_REGISTER_API_KEY generated by setup_server.sh (from Docker/.env on the server).

    your_generated_api_key_from_server
    

• (Required for HTTPS) Server Certificate:

  • Copy the public certificate of your pi-registrar service from the server to the Pi.
  • Source on server: AeroScan/Docker/config/certs/cert.pem
  • Destination on Pi: /etc/ssl/certs/pi_registrar_server.pem (this path is referenced in main.py).
  • Manual Copy Example (run from server):

    scp ~/AeroScan/Docker/config/certs/cert.pem <pi_user>@<pi_ip>:/tmp/cert.pem
    

  • Then on the Pi:

    sudo mv /tmp/cert.pem /etc/ssl/certs/pi_registrar_server.pem
    sudo chown root:root /etc/ssl/certs/pi_registrar_server.pem
    sudo chmod 644 /etc/ssl/certs/pi_registrar_server.pem
    

    (This step is a prime candidate for automation with Ansible later).

• (Optional) RPi Connect Device Name (if using that Grafana link feature):

  • File: /var/local/rpi_connect_device_name.txt
  • Content: The name you assigned this Pi in your connect.raspberrypi.com dashboard.

    MyLivingRoomPi
    

3. main.py Execution: Your image's boot process should:

1. Navigate to the cloned AeroScan/pi/software/ directory.
2. Execute python3 main.py. It's highly recommended to run main.py as a systemd service for reliability and auto-restarts. Create a file like /etc/systemd/system/aeroscan.service:

[Unit]
Description=AeroScan Network Monitor
Wants=network-online.target
After=network-online.target git-pull-aeroscan.service # Example: ensure network is up and git pull (if separate service) ran

[Service]
User=aeroscan # Or 'pi', the user running the script
Group=aeroscan # Or 'pi'
WorkingDirectory=/home/aeroscan/AeroScan/pi/software # Adjust to your clone path
ExecStart=/usr/bin/python3 /home/aeroscan/AeroScan/pi/software/main.py
Restart=always
RestartSec=10
# Optional: StandardOutput=journal, StandardError=journal for logging

[Install]
WantedBy=multi-user.target

Then enable and start it: sudo systemctl enable aeroscan.service && sudo systemctl start aeroscan.service. (You might need a separate service/script for the git pull part if it's complex or needs to happen before this service starts).

C. Ansible for Configuration Management (Setup on Central Server)

1. Install Ansible (if not already done during server prerequisites):

   sudo apt update && sudo apt install -y ansible
   

2. Prepare SSH Key-Based Access to Pis:

   • On your central server (it06), ensure you have an SSH key pair (~/.ssh/id_rsa, ~/.ssh/id_rsa.pub). If not, run ssh-keygen.
   • Copy the server's public key to each Raspberry Pi for the user Ansible will connect as (e.g., aeroscan):

     # On server:
     ssh-copy-id aeroscan@<pi_ip_or_hostname>
     

3. Dynamic Inventory Script (AeroScan/ansible/inventory_from_registrar.py):

   • This script is included in the repository.
   • Ensure it's executable: chmod +x ~/AeroScan/ansible/inventory_from_registrar.py.
   • It expects the pi-registrar's public certificate at ~/AeroScan/ansible/config/pi_registrar_server.pem (this is copied by setup_server.sh).
   • It uses https://AeroScan:5001/targets by default. Ensure AeroScan resolves to your server's IP on the Ansible control node (e.g., add <your_server_ip> AeroScan to /etc/hosts on the control node if not a public DNS name).

4. Ansible Configuration (AeroScan/ansible/ansible.cfg): This file should be present in AeroScan/ansible/ and point to your dynamic inventory:

   [defaults]
   inventory         = ./inventory_from_registrar.py
   host_key_checking = False ; For trusted local networks; set to True for stricter security.
   deprecation_warnings = False
   
   [privilege_escalation]
   become            = True
   become_method     = sudo
   # become_ask_pass = False ; Assumes passwordless sudo is configured on Pis for the ansible_user
   

5. Passwordless Sudo on Pis for Ansible User: For Ansible to manage system settings (like nmcli for WiFi), the user it connects as (e.g., aeroscan) needs passwordless sudo rights on the Pis for specific commands. Edit /etc/sudoers on each Pi using sudo visudo:

   • Example for user aeroscan needing nmcli and systemctl: aeroscan ALL=(ALL) NOPASSWD: /usr/bin/nmcli, /bin/systemctl
   • Or for all commands (less secure): aeroscan ALL=(ALL) NOPASSWD: ALL

6. Run Ansible Playbooks: Navigate to AeroScan/ansible/ and run playbooks:

   cd ~/AeroScan/ansible/
   ansible-playbook playbooks/your_playbook_name.yml
   

   (Start with playbooks/test_create_file.yml to confirm setup).

Usage

• Central Server: The Docker services (Prometheus, Grafana, pi-registrar) run continuously.
• Raspberry Pi Nodes: The main.py script collects and exposes metrics. It registers with pi-registrar.
• Monitoring: Access Grafana at http://<your_server_ip>:3000 to view dashboards.
• Configuration: Use Ansible playbooks from the central server to manage Pis.
• Troubleshooting:
  • Check docker-compose logs <service_name> on the server.
  • Check main.py output or systemd service logs (journalctl -u aeroscan.service if set up) on the Pis.
  • Use Ansible ad-hoc commands or playbooks for diagnostics.

License

AeroScan is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0).

You are free to share (copy and redistribute the material in any medium or format) and adapt (remix, transform, and build upon the material) under the following terms:

• Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
• NonCommercial — You may not use the material for commercial purposes.
• ShareAlike - If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.
• No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.

For more information, see the full license text in this repository or visit: https://creativecommons.org/licenses/by-nc-sa/4.0/