Graph Resources Using Gnuplot on Linux

My previous post dealt with retrieving some data and then dumping it into a csv file. Here is a script and gnuplot configs that creates an index page. The script finds all the csv files in the specified $DIRECTORY. For every csv file found, a new *_summary directory is created with an index.html files, and the png graph images.

This is the main script file. Make sure to adjust the DIRECTORY and GRAPHDIR variables for your environment.

create_graph.sh

       
#!/bin/bash

#directory that contains the csv files
DIRECTORY=/dashcam/resourcelog_archive
#directory that contains the gnuplot config files
GRAPHDIR=/dashcam/graphs

for f in $( ls "$DIRECTORY"/*.csv ); do
        HTMLDIR=$(echo "$f" | sed 's/.csv//g')
        #echo HTMLDIR is "$HTMLDIR"

        mkdir -vp "$HTMLDIR"_summary

        echo "$f" > "$DIRECTORY"/gnuplot_variable.txt

        /usr/bin/gnuplot -e "filename='$f'" "$GRAPHDIR"/temperature.gp > "$HTMLDIR"_summary/temperature.png
        /usr/bin/gnuplot -e "filename='$f'" "$GRAPHDIR"/memory.gp > "$HTMLDIR"_summary/memory.png
        /usr/bin/gnuplot -e "filename='$f'" "$GRAPHDIR"/cpu.gp > "$HTMLDIR"_summary/cpu.png

        #Create webpage
        echo "<center>"$HTMLDIR"</center><br><br>" > "$HTMLDIR"_summary/index.html
        echo "<center><img src="cpu.png"></center><br><br>" >> "$HTMLDIR"_summary/index.html
        echo "<center><img src="memory.png"></center><br>" >> "$HTMLDIR"_summary/index.html
        echo "<center><img src="temperature.png"></center><br>" >> "$HTMLDIR"_summary/index.html

done

#Create index webpage
echo "<center><b>Index</b></center><br><br>" > "$DIRECTORY"/index.html

for d in $( ls -d "$DIRECTORY"/*_summary | xargs -n 1 basename ); do
        echo  "<a href=$d>$d</a><br>" >> "$DIRECTORY"/index.html
done
       

All the *.gp files should be in $GRAPHDIR. All graphs are sized at 1000x400.

This template graphs 1, 5, and 15 minute CPU load over time.
cpu.gp

       

#!/usr/bin/gnuplot
reset
set terminal pngcairo size 1000,400
set datafile separator ","

set xdata time
set timefmt "%Y-%m-%d,%H:%M:%S"
set format x "%H:%M"

set xlabel "Time (Hour:Minute GMT)"
set ylabel "CPU"

set title "CPU"
set key below
set style data line
set grid

plot filename using 1:4 title "1 min", filename using 1:5 title "5 min", filename using 1:6 title "15 min"

       

This template graphs Free, Cached, and Total Memory.
memory.gp

       

#!/usr/bin/gnuplot
reset
set terminal pngcairo size 1000,400
set datafile separator ","

set xdata time
set timefmt "%Y-%m-%d,%H:%M:%S"
set format x "%H:%M"

set xlabel "Time (Hour:Minute GMT)"
set ylabel "Memory (KB)"

set title "Memory"
set key below
set style data line
set grid

plot filename using 1:8 title "Free", filename using 1:7 title "Total", filename using 1:9 title "Cached"

       

This template graphs degrees celcius over time.
temperature.gp

       

#!/usr/bin/gnuplot
reset
set terminal pngcairo size 1000,400
set datafile separator ","

set xdata time
set timefmt "%Y-%m-%d,%H:%M:%S"
set format x "%H:%M"

set xlabel "Time (Hour:Minute GMT)"
set ylabel "Celcius"

set title "CPU Temperature"
set key below
set style data line
set grid

plot filename using 1:3 title "Degrees Celcius"

       

Raspberry Pi Resource Monitor

There are plenty of resource monitors and graph utilities out there for the Raspberry Pi. RPi-Monitor is a great one. However, I wanted to learn a little more about how to collect data for the various resources, so I made my own simple bash script. Here are some basic commands that can be stringed together to monitor the basic resources of the Raspberry Pi. I output the data to a csv file that can be graphed using gnuplot.

Skip to the bottom for the full script.

First, create a new file using text editor.

nano resourcelog.sh

Start with the basics. Tell the shell to use bash.

#!/bin/bash

Most of the time I am getting time from a GPS dongle. Wait a minute to allow a GPS lock.

sleep 60

Create a date/time variable used to name the log file. I do this so that every time the Pi is powered on, it creates a new file with a name based on the date/time the logging was started.

LABELDATE=$(date +"%Y-%m-%d_%H:%M")

Make a directory to put the log files.

mkdir -vp /home/pi/resourcelog_archive/

Name the columns in the csv file. You can see we will be logging date, time, degrees celcius of the pi CPU, average CPU loads, free memory, cached memory, swap total, and swap free. This gets put into a new file in the resourcelog_archive directory, using the date variable defined above. (The command should be one line)

echo "Date,Time,Degrees Celcius,CPU 1 Min,CPU 5 min,CPU 15 min,Memory,Free Memory,Cached Memory,Swap Total,Swap Free" > /home/pi/resourcelog_archive/resourcelog_"$LABELDATE".csv

Create a while loop that will query for the info specified. I do this by defining a variable ($CYCLE) and the loop runs while the variable equals the original value (1).

CYCLE=1
while  [ $CYCLE -eq 1 ]
do

Create variables for data gathering. These variables are part of the while loop and are updated every 60 seconds.

Define the current date in YY-MM-DD format.

DATESTAMP=$(date +"%Y-%m-%d")

Define the current time in HH:MM format.

TIMESTAMP=$(date +"%H:%M")

Define the temperature of the APU (CPU) in degrees celcius. This command uses vcgencmd measure_temp, shows only data after the = symbol, and then removes the "'C" so we are left with only a decimal number.

TEMP=$(/opt/vc/bin/vcgencmd measure_temp | cut -d '=' -f 2 | sed s/\'C//g)

Define the CPU load average. This runs the loadavg command, only shows the last three fields, and then replaces the spaces with commas, giving an output formatted as 0.0,0.0,0.0

CPU=$(cat /proc/loadavg | cut -d ' ' -f -3 | sed 's/ /,/g')

Define the total amount of memory in kB. This uses /proc/meminfo, finds "MemTotal", displays only column 2, then removes the "kB" so only numbers are output.

 MEMTOTAL=$(cat /proc/meminfo | egrep MemTotal | awk '{print $2}' | sed 's/kB//g')

Define the total amount of free memory. Similar to above, but searches for "MemFree".

MEMFREE=$(cat /proc/meminfo | egrep MemFree | awk '{print $2}' | sed 's/kB//g')

Define the total amount of cached memory. Similar to above, but searches for only matches that start with Cached. (^Cached).

CACHED=$(cat /proc/meminfo | egrep '^Cached' | awk '{print $2}' | sed 's/kB//g')

Define the swap file allocation. Similar to above, but searches for "SwapTotal".

SWAPTOTAL=$(cat /proc/meminfo | egrep SwapTotal | awk '{print $2}' | sed 's/kB//g')

 Define the amount of swap available (unused).

SWAPFREE=$(cat /proc/meminfo | egrep SwapFree | awk '{print $2}' | sed 's/kB//g')

That's all the resources I monitor. Now the variables need to be written to the file created at the beginning of the script. This just writes a new line at the end of the file with the variable specified above seperated by commas.

echo "$DATESTAMP","$TIMESTAMP","$TEMP","$CPU","$MEMTOTAL","$MEMFREE","$CACHED","$SWAPTOTAL","$SWAPFREE" >> /home/pi/resourcelog_archive/resourcelog_"$LABELDATE".csv

 Wait for 60 seconds and then start again.

sleep 60

Close the loop.

done

 Save the file and exit. Make the file executable.

chmod +x resourcelog.sh

Make the file run on every reboot. This assumes you are logged in as the user "pi".

crontab -e
@reboot /home/pi/resourcelog.sh

That's it, the pi will run the resourcelog.sh script every time it's powered on, and record the results in a new file in the resourcelog_archive directory.  Feel free to tell me there's a better way to do this, I'm sure there is something more efficient. The next post will go over creating graphs automatically using gnuplot.

Here is the full code:

       
#!/bin/bash
# Wait for GPS time
sleep 60

LABELDATE=$(date +"%Y-%m-%d_%H:%M")
mkdir -vp /home/pi/resourcelog_archive/
echo "Date,Time,Degrees Celcius,CPU 1 Min,CPU 5 min,CPU 15 min,Memory,Free Memory,Cached Memory,Swap Total,Swap Free" > /home/pi/resourcelog_archive/resourcelog_"$LABELDATE".csv
CYCLE=1
while  [ $CYCLE -eq 1 ]
do

DATESTAMP=$(date +"%Y-%m-%d")
TIMESTAMP=$(date +"%H:%M")
TEMP=$(/opt/vc/bin/vcgencmd measure_temp | cut -d '=' -f 2 | sed s/\'C//g)
CPU=$(cat /proc/loadavg | cut -d ' ' -f -3 | sed 's/ /,/g')
MEMTOTAL=$(cat /proc/meminfo | egrep MemTotal | awk '{print $2}' | sed 's/kB//g')
MEMFREE=$(cat /proc/meminfo | egrep MemFree | awk '{print $2}' | sed 's/kB//g')
CACHED=$(cat /proc/meminfo | egrep '^Cached' | awk '{print $2}' | sed 's/kB//g')
SWAPTOTAL=$(cat /proc/meminfo | egrep SwapTotal | awk '{print $2}' | sed 's/kB//g')
SWAPFREE=$(cat /proc/meminfo | egrep SwapFree | awk '{print $2}' | sed 's/kB//g')


echo "$DATESTAMP","$TIMESTAMP","$TEMP","$CPU","$MEMTOTAL","$MEMFREE","$CACHED","$SWAPTOTAL","$SWAPFREE" >> /home/pi/resourcelog_archive/resourcelog_"$LABELDATE".csv
sleep 60
done