FreedomBox for Communities/Monitoring Server
The monitoring client on the FreedomBox server of each of the communities will need to send the collected metrics to a central location. This central location, the monitoring server, needs to store the data sent, visualize the data for administrator and alert them when something is off. This section describes the setup on the server. It is assumed that the server is also running a FreedomBox. However, with minor changes to the instructions any server running Debian GNU/Linux Buster or above can be used as a server.
Setup Graphite Carbon
[edit | edit source]Graphite Carbon server is responsible collecting the data from the various FreedomBox servers and storing them in a time-series database called Whisper. Run the following commands to setup Graphite Carbon.
- Install Graphite Carbon.
apt install graphite-carbon
- Open up ports in the firewall for receiving data from FreedomBox Servers.
firewall-cmd --zone=external --add-port=2003/tcp firewall-cmd --zone=external --add-port=2003/tcp --permanent
- After this, graphite-carbon server will be running and listening on port 2003. This can be checked using the following commands:
systemctl status carbon-cache ss -nlpt | grep carbon-cache
Setup Graphite Web
[edit | edit source]Graphite Web service is responsible for serving the data stored by Graphite Carbon to consumers that can visualize the data such as Grafana. Run the following commands to setup Graphite Web.
- Install Graphite Web
apt install graphite-web libapache2-mode-wsgi-py3
- Generate a secret key for Django framework used by Graphite Web.
python3 -c 'import random; result = "".join([random.choice("abcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*(-_=+)") for i in range(64)]); print(result)'
- Note the key generated from previous command and set it as secret key in the file /etc/graphite/local_settings.py.
SECRET_KEY="<output from the key generation command>"
- Create a database for Graphite Web to store its own data.
sudo --user _graphite graphite-manage migrate --run-syncdb chown _graphite:_graphite /var/lib/graphite/graphite.db chmod 0600 /var/lib/graphite/graphite.db
- Create Apache configuration for graphite-web and start serving.
cat << EOF > /etc/apache2/sites-available/graphite-web.conf Listen 8080 <VirtualHost *:8080> WSGIDaemonProcess _graphite processes=5 threads=5 display-name='%{GROUP}' inactivity-timeout=120 user=_graphite group=_graphite WSGIProcessGroup _graphite WSGIImportScript /usr/share/graphite-web/graphite.wsgi process-group=_graphite application-group=%{GLOBAL} WSGIScriptAlias / /usr/share/graphite-web/graphite.wsgi Alias /static/ /usr/share/graphite-web/static/ <Location "/static/"> SetHandler None </Location> </VirtualHost> EOF a2ensite graphite-web systemctl restart apache2
Setup Grafana
[edit | edit source]Grafana consumes data from Graphite Web and produces beautiful visualizations. It can also be configured for alerting when certain event occurs in the data. Run the following commands to setup Grafana.
- Install Grafana from it's project's Debian repositories.
apt-get install -y apt-transport-https wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add - cat << EOF > /etc/apt/sources.list.d/grafana.list deb https://packages.grafana.com/oss/deb stable main EOF apt update apt install grafana
- Configure Grafana to be served from the a existing domain under a sub-path. Edit the file /etc/grafana/grafana.ini and ensure the following two directives are configured as shown.
root_url = https://<domain_name>/monitor/ serve_from_sub_path = true
- Configure Grafana to start every time the machine is booted and start Grafana.
systemctl daemon-reload systemctl enable grafana-server.service systemctl start grafana-server.service
- Create Apache configuration to proxy Grafana from a sub-path and start serving.
cat << EOF > /etc/apache2/conf-available/grafana.conf <Location /monitor/> ProxyPass http://localhost:3000/monitor/ </Location> EOF a2enconf grafana systemctl reload apache2
- Setup credentials for Grafana access. Visit https://<domain_name>/monitor/ and login with the following credentials. You will be prompted to change the password. Set a strong password.
- Username: admin
- Password: admin
- Create user accounts for each for users who will using the monitoring setup. In the left menu, go to Server Admin, Users. Click on the New User button and provide details such as name, email, username and password to create a new user account.
- Connect Grafana to the Graphite Carbon database via Graphite Web we have configured in the previous sections. Go to Configuration, Data Sources and select Carbon data source. In the configuration dialog that shows up, provide the following details:
- Name: Carbon (or anything you choose)
- URL: http://localhost:8080
- Access: Server
- Version: 1.1
Creating a Dashboard for Each Community
[edit | edit source]In Grafana interface, you need to create a dashboard for monitoring each of the communities you are the administering. The following instructions provide an overview of how this can be done:
- Login to Grafana interface.
- In the menu, select Create, Dashboard. A dashboard is created and a prompt is shown to add a new panel. Select a Graph.
- A new dashboard is created with a simple graph.
- Save dashboard by clicking the save icon. In the save dialog, provide a name for the dashboard based on the community for which the dashboard is being created. Click Save.
- To start showing proper data in the panel, click on Panel Title and select Edit. In the Metric tab, click on select metric drop down. You should see the name of the domain you assigned to your community as configured in collectd. Then select a metric such as cpu-0 and then cpu-user. You should start seeing the plot of the metric in the graph. Save the dashboard.
Parameters to Monitor
[edit | edit source]It is recommended that you create a dashboard to your taste but monitor the following parameters at least for each community:
- CPUs
- Series: <community> • * • cpu-user
- Functions: averageSeries() • alias(User)
- Series: <community> • * • cpu-nice
- Functions: averageSeries() • alias(Nice)
- Series: <community> • * • cpu-system
- Functions: averageSeries() • alias(System)
- Series: <community> • * • cpu-wait
- Functions: averageSeries() • alias(Wait)
- Memory
- Series: <community> • memory • percent-used
- Functions: alias(% Used)
- Load
- Series: <community> • load • load • shorterm
- Functions: alias(5min)
- Series: <community> • load • load • midterm
- Functions: alias(10min)
- Series: <community> • load • load • longterm
- Functions: alias(15min)
- Disk
- Series: <community> • <disk> • disk_ops • read
- Functions: alias(Read ops)
- Series: <community> • <disk> • disk_ops • write
- Functions: alias(Write ops)
- Series: <community> • <disk> • disk_octets • read
- Functions: alias(Read Bytes)
- Series: <community> • <disk> • disk_octets • write
- Functions: alias(Write Bytes)
- Disk I/O Time
- Series: <community> • <disk> • pending_operations
- Functions: alias(Pending Operations)
- Series: <community> • <disk> • disk_io_time • io_time
- Functions: alias(I/O Time)
- Series: <community> • <disk> • disk_io_time • weighted_io_time
- Functions: alias(Weighted I/O Time)
- <mountpoint> Partition (for each partition)
- Series: <community> • df-<mountpoint> • percent_bytes-used
- Functions: alias(Used)
- Series: <community> • df-<mountpoint> • percent_bytes-reserved
- Functions: alias(Reserved)
- <LAN/WAN> Network Bandwidth (for each interface)
- Series: <community> • interface-<interface> • if_octets • rx
- Functions: alias(Received)
- Series: <community> • interface-<interface> • if_octets • tx
- Functions: alias(Transmitted)
- Connections Tracked
- Series: <community> • conntrack • conntrack
- Functions: alias(Connections Tracked)
- Wi-Fi Access Points
- alias(sumSeries(isNonNull(<community>.ping.ping-10_42_0_[1-9][0-9])), 'Wi-Fi Access Points')
- Internet Uplink
- alias(isNonNull(<community>.ping.ping-10_42_0_4), 'Internet Router')
- alias(isNonNull(<community>.ping.ping-10_42_0_5), 'PTP Link - Router A')
- alias(isNonNull(<community>.ping.ping-10_42_0_6), 'PTP Link - Router B')
- Internet Ping
- Series: <community> • ping • ping-1_1_1_1
- Functions: alias(Internet Ping)
- Processes
- Series: <community> • processes • ps_state-running
- Functions: alias(Running)
- Series: <community> • processes • ps_state-sleeping
- Functions: alias(Sleeping)
- Series: <community> • processes • ps_state-paging
- Functions: alias(Paging)
- Series: <community> • processes • ps_state-blocked
- Functions: alias(Blocked)
- Series: <community> • processes • ps_state-zombies
- Functions: alias(Zombies)
- Series: <community> • processes • ps_state-stopped
- Functions: alias(Stopped)
- Uptime
- Series: <community> • uptime • uptime
- Functions: alias(Time Since Reboot)
- Users
- Series: <community> • users • users
- Functions: alias(Logged-In Users)
- collectd Queue
- Series: <community> • collectd-cache • cache_size
- Functions: alias(Cache Size)
- Series: <community> • collectd-write_queue • derive-dropped
- Functions: alias(Dropped Metrics)
- Series: <community> • collectd-write_queue • queue_length
- Functions: alias(Queue Length)