The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.

NAME

Wubot::Guide::ArduinoSensors - monitoring arduino sensor data

DESCRIPTION

This document describes how to read temperature, humidity, or other sensor data from an Arduino and feed it into wubot. We are notified in the event of a significant change in the sensor data or when the sensor data gets outside of some configured thresholds. The data is stored in a round-robin database (RRD) and daily, weekly, and monthly graphs are generated from the data. Custom graphs are also illustrated. There is also some discussion of monitoring a wireless sensor network built with XBee.

Arduino sensor data

I won't discuss the process of wiring up a temperature sensor or programming an arduino here, as that is already covered in great detail elsewhere.

Here are some temperature sensors I can recommend:

  - http://shop.moderndevice.com/products/tmp421-temperature-sensor
  - http://www.sparkfun.com/products/9569
  - http://vegetronix.com/Products/VG400/

Any sensor that will work with the arduino will work fine. wubot actually reads the sensor data over the serial port, so if the ardunino can read the data and send it over the serial port, then wubot can handle it.

The format of the data used here is CSV with four fields:

  ^source, type, value, units

Here are a few examples.

  ^office, temp, 75, F
  ^outside, humidity, 80, percent
  ^garden, moisture, 60, percent

Note that there is a carrot, '^', here at the beginning of the line to indicate the beginning of a new record. When an arduino is reset, a bit of junk often comes through at the beginning. The beginning-of-line field allows us to filter out any junk at the beginning of the line.

You can really use any format that you want for the data, but the reactor rules below assume that it will start in this format.

You will probably want to send the sensor information every 1 to 5 minutes. It is not recommended that you send it more often than once every 15 seconds.

I am still learning arduino programming, but you might be able to find some useful examples in my farmbot project here:

  http://github.com/wu/farmbot

SerialPort monitor

By the time you get to this point, you should have the arduino hooked up to the sensors and should be sending the data over the serial port in the format:

  ^source, type, value, units

To get the data into wubot, start by setting up an instance of the SerialPort monitor to read from your serial port. The config file would live here:

  ~/wubot/config/plugins/SerialPort/labduino.yaml

And here is an example:

  ---
  delay: 5s
  device: /dev/tty.usbmodem3b11

This will check the configured device every 5 seconds for sensor data. Each line of data will be put into a message. The 'line' field will contain the string that was read.

parsing the message

Now that you have the sensor data coming in, the next step is to parse it. To do that we need to define some rules. See also Wubot::Guide::Rules.

Remember that 'beginning-of-record' character I mentioned above? This rule makes use of that. Start by cleaning everything from the beginning of the line up until the beginning-of-record character:

  - name: clean to beginning of line
    plugin: TransformField
    config:
      source_field: line
      regexp_search: '^.*\^'

At this point I find it useful to display the line that was read to stdout in the terminal running the monitor process:

  - name: dump line
    plugin: Dumper
    config:
      field: line

Next, assuming the format above is being used for the data, we can split the CSV data into the four fields:

  - name: split
    plugin: Split
    config:
      source_field: line
      target_fields:
        - source
        - type
        - value
        - units

Now that the data is split, let's check that we have a valid record, i.e. all the fields got parsed. Any that don't get routed to the bit bucket using the 'last_rule' flag.

      - name: get rid of invalid fields
        condition: source is false OR type is false OR value is false OR units is false
        last_rule: 1

This rule is just a bit trickier. Right now we have the data in four fields. For example:

    source: office
    type: temp
    value: 78.3
    units: F

It would really be more convenient at this point to have a 'temp' field on the message that was set to '78.3'. That will make creating some of the later rules a bit simpler. So here we can use the CopyField plugin to take the value (78.3) and store it in a field that is named in the 'type' field (temp). Here is the rule:

  - name: map key and value names
    plugin: CopyField
    config:
      source_field: value
      target_field_name: type

And after this rule, the message will contain the new field:

    source: office
    type: temp
    value: 78.3
    units: F
    temp: 78.3

Since all of this is really simple stuff that just involves shuffling around some data on the message, we'll do all of this directly in the monitor config. Here is the resulting config:

  ---
  delay: 5s
  device: /dev/tty.usbmodem3b11

  react:

    - name: lines
      condition: line is true

      rules:

        - name: clean to beginning of line
          plugin: TransformField
          config:
            source_field: line
            regexp_search: '^.*\^'

        - name: dump line
          plugin: Dumper
          config:
            field: line

        - name: split
          plugin: Split
          config:
            source_field: line
            target_fields:
              - source
              - type
              - value
              - units

        - name: get rid of invalid fields
          condition: source is false OR type is false OR value is false OR units is false
          last_rule: 1

        - name: map key and value names
          plugin: CopyField
          config:
            source_field: value
            target_field_name: type

Alerts

Up until this point we've just been focused on getting the data into the system. Now it's time to set up the reactor config.

Start building a rule tree where the parent rule looks for messages coming form the serial port that have sensor data.

  - name: arduino
    condition: key matches SerialPort AND source is true
    rules:

We'll start by doing something a bit unusual. We'll create a custom 'key' for this sensor reading. Normally the 'key' message is set to the name of the plugin and the name of the instance config file. This is great if you have a separate config file for each feed. But there is only one serial port and there may be many different sensors. So we're going to start by creating a custom key that contains the sensor source and type, e.g. 'office-temp'. This just makes it a lot easier for later plugins keep track of the office temp and the outside temp as two separate values. Note that this change to the 'key' field is permanent! So any later rules in the reactor that look for 'key matches SerialPort' will not find these messages again!

      - name: key
        plugin: Template
        config:
          template: '{$source}-{$type}'
          target_field: key

Let's start with a simple warning. If you haven't looked at the Wubot::Guide::Notifications doc yet, this might be a good time to take a look. If the temp gets above 85 degrees in the office, then generate an alert message. There are computers in the office, so if the temerature starts to get too high, the one of them could overheat. So let's look for any message that indicates a temperature of greater than 85 and set a few fields on the message. The 'subject' field will be used by notification plugins such as Growl and Console. The 'sticky' field is used by Growl to cause the message to stick on the screen until it is dismissed. The 'color' field can be used to colorize the notifications. As long as we're in there, we'll set a 'heat_warning' field in case we want to use this in a future reaction.

          - name: temp high
            condition: temp > 85
            plugin: SetField
            config:
              set:
                heat_warning: 1
                color: yellow
                sticky: 1
                subject: warning: temp is high

Thanks to the 'subject' field, we can get a notification that the temperature is high. But the message will simply say 'warning: temp is high', it won't actually tell us what the temperature is. For that we can use the 'Template' plugin, which creates a field using a template. Any params in the template are filled from other fields on the message. So here is a heat warning notification that will actually inclue the temperature sensor reading:

          - name: heat warning
            condition: heat_warning is true
            plugin: Template
            config:
              target_field: subject
              template: warning: temperature is high: {$temp}

Now when the temperature is above 85 degrees in the office, I'll get a sticky notification telling me the current temperature. Then I'll go upstairs and open the windows or possibly turn on the fan.

Beyond just tracking when the sensor reading goes outside of a configured threshold, we also want to track the change in sensor readings over time. For that there is a 'State' plugin, which caches the previously seen sensor readings and can send an alert when the value changes by more than a certain threshold. You can specify that you want to be notified if the value changes in either directory, or only if it increases or decreases.

For example, suppose we want to get a notification any time the soil moisture drops more than 10 percent. The following rule tree would select messages that have a 'moisture' field, and then use the 'State' plugin to track the 'moisture' field and send a notification if the value decreases by more than 10. The 'State' plugin will set a 'subject' on the message that will indicate the current moisture level, amount of time since the last sensor change, and the highest value during that time period. It will also set the 'state_changed' flag on the message. This is pretty important (it usually means the garden is starting to get dry), so we'll make sure the alert gets the 'sticky' flag so it will stay on the screen and get my attention.

      - name: moisture
        condition: contains moisture
        rules:

          - name: moisture drop
            plugin: State
            config:
              field: moisture
              decrease: 10

          - name: moisture drop sticky
            condition: state_changed is true
            plugin: SetField
            config:
              field: sticky
              value: 1

We also want to monitor for temperature changes, just to have an idea about how the temperature is changing. These aren't as important, so there's no sticky flag here:

      - name: temp
        condition: contains temp
        rules:
          - name: temp change
            plugin: State
            config:
              field: temp
              change: 5

The State plugin will send a warning if you stop receiving data for any of the tracked datasets.

RRD and graphs

It's nice to get alerts when the values change or get outside of safe thresholds, but we also want to preserve historical data and generate some graphs. This is where the RRD plugin comes in hand. Here's an example RRD plugin that will take the 'value' and jam it into an RRD file and generate a daily, weekly, and monthly graph:

      - name: rrd
        condition: source is true AND type is true AND value is true AND units is true
        plugin: RRD
        config:
          base_dir: /home/wu/wubot/rrd
          fields:
            value: GAUGE
          step: 60
          period:
            - day
            - week
            - month
          graph_options:
            right-axis: 1:0
            width: 375

The RRD file will get written to:

  ~/wubot/rrd/rrd/{key}

and the graphs will be generated to:

  ~wubot/rrd/graphs/{key}

Graphs with multiple sources

Here are some example graphs:

  - http://www.geekfarm.org/wu/wubot/Coopduino.now.png
  - http://www.geekfarm.org/wu/wubot/Coopduino.png
  - http://www.geekfarm.org/wu/wubot/Coopduino-week.png
  - http://www.geekfarm.org/wu/wubot/Growbot.png

SQLite

Perhaps you have some other ideas for analyzing the sensor data. Well, just drop it in a SQLite database:

      - name: sqlite
        condition: value is true
        plugin: SQLite
        config:
          file: /home/wu/wubot/sqlite/arduino.sql
          tablename: sensors
          schema:
            id: INTEGER PRIMARY KEY AUTOINCREMENT
            source: int
            type: int
            value: int
            units: int
            lastupdate: int

XBee and wireless sensor networks