Development of the Internal Temperature and Humidity Sensor

Details

This is the sensor that will be placed inside the brood box of the hive and will be used to monitor the internal temperature and humidity of the hive.
This sensor will be mounted into the inside wall of the hive.
Under normal circumstances the bees maintain a constant temperature and humidity inside the brood box.  In a healthy beehive, the bees are really good at keeping this constant, so changes to these norms could be an indicator that something is wrong within the beehive.

This is going to be harder than i thought...

On Sunday I started with this circuit, I took the designs from this guide from Adafruit.  Given that this is a complete guide on how to make this work I thought it was going to be an easy starting point, boy was i wrong...

I followed the guide and built this sensor, it looks like a mess, but is only a prototype, so function over beauty it is...
On Sunday i spent several hours trying to read this sensor using the Raspberry Pi and Python with little success, i rebuilt this breadboard several times, replacing all the components incase something was faulty and still i could not get any readings.  I did make it a little harder than i needed to as i was stubborn and wanted to learn how to read from this sensor myself rather than using the Adafruit library in the guide, this did help me to learn a lot...
After several hours and much time reading through various articles and guides I came across a few mentions that the DHT11 sensor when using a Raspberry Pi and Python as the programming language can be problematic, people had had better success using C, but it was still not 100% accurate.  On further reading this appeared to be because the kernel in linux doesn't give enough processing time to Python or C in order for the data to be received from the sensor.  The sensor sends a blob of data when asked to provide a reading.
I happened to have an Arduino to hand so i connected it up to the sensor that i had already built to see if i got any better results from that, and I did.  So i now knew that my circuit was good and it was definitely something to do with the scheduling of the processes, this was a great relief as i then knew that my sensor and circuit worked, but i still had the issue of getting it to work with the Raspberry Pi.
I finally solved this problem when i found an article that people were using Arduino's and Raspberry Pi's in combination with great success.  Basically they used the Ardunio specifically to collect data from the sensor and then the Raspberry Pi would read the data from the Arduinos USB port.  Here us a great article explaining it all.
I then built a simpler circuit for the Arduino, and connected the sensor, the Arduino and the Raspberry pi together, and i ended up with this.

I started off with a simple C program on the Arduino that collected the sensor information and outputted it to the screen, I don't know C so i started off with a piece of code that i downloaded from here and uploaded it to the Arduino.
I then used my raspberry pi to read the data from the Arduino, by treating it as a COM port.
This showed me that i was able to receive data to the Raspberry Pi from the Arduino and this was a good achievement, but the output was not what i wanted, i decided i was going to make the Arduino output in a format called JSON, this is a nice easy to read structured data format, so i re-wrote the C program that i got from UU gear and turned it into this:
// 
//   FILE:  dht11_test1.pde
// PURPOSE: DHT11 library test sketch for Arduino
//

//Celsius to Fahrenheit conversion
double Fahrenheit(double celsius)
{
return 1.8 * celsius + 32;
}

// fast integer version with rounding
//int Celcius2Fahrenheit(int celcius)
//{
//  return (celsius * 18 + 5)/10 + 32;
//}


//Celsius to Kelvin conversion
double Kelvin(double celsius)
{
return celsius + 273.15;
}

// dewPoint function NOAA
// reference (1) : http://wahiduddin.net/calc/density_algorithms.htm
// reference (2) : http://www.colorado.edu/geography/weather_station/Geog_site/about.htm
//
double dewPoint(double celsius, double humidity)
{
// (1) Saturation Vapor Pressure = ESGG(T)
double RATIO = 373.15 / (273.15 + celsius);
double RHS = -7.90298 * (RATIO - 1);
RHS += 5.02808 * log10(RATIO);
RHS += -1.3816e-7 * (pow(10, (11.344 * (1 - 1/RATIO ))) - 1) ;
RHS += 8.1328e-3 * (pow(10, (-3.49149 * (RATIO - 1))) - 1) ;
RHS += log10(1013.246);

        // factor -3 is to adjust units - Vapor Pressure SVP * humidity
double VP = pow(10, RHS - 3) * humidity;

        // (2) DEWPOINT = F(Vapor Pressure)
double T = log(VP/0.61078);   // temp var
return (241.88 * T) / (17.558 - T);
}

// delta max = 0.6544 wrt dewPoint()
// 6.9 x faster than dewPoint()
// reference: http://en.wikipedia.org/wiki/Dew_point
double dewPointFast(double celsius, double humidity)
{
double a = 17.271;
double b = 237.7;
double temp = (a * celsius) / (b + celsius) + log(humidity*0.01);
double Td = (b * temp) / (a - temp);
return Td;
}


#include <dht11.h>

dht11 DHT11;

#define DHT11PIN 2

void setup()
{
  Serial.begin(115200);
}

void loop()
{

  int chk = DHT11.read(DHT11PIN);

  String jsonData;

  char humidity_tmp[10];
  dtostrf((float)DHT11.humidity,1,2,humidity_tmp);
  String humidity_tmp_str;
  humidity_tmp_str = String(humidity_tmp);
  char temp_C_tmp[10];
  dtostrf((float)DHT11.temperature,1,2,temp_C_tmp);
  String temp_C_tmp_str;
  temp_C_tmp_str = String(temp_C_tmp);
  char temp_F_tmp[10];
  dtostrf(Fahrenheit(DHT11.temperature),1,2,temp_F_tmp);
  String temp_F_tmp_str;
  temp_F_tmp_str = String(temp_F_tmp);
  char temp_K_tmp[10];
  dtostrf(Kelvin(DHT11.temperature),1,2,temp_K_tmp);
  String temp_K_tmp_str;
  temp_K_tmp_str = String(temp_K_tmp);

  //Serial.print("Read sensor: ");
  switch (chk)
  {
    case DHTLIB_OK: 
jsonData = "{ \"humidity\":" + humidity_tmp_str + ", \"temp_C\":" + temp_C_tmp_str + ", \"temp_F\":" + temp_F_tmp_str + ", \"temp_K\":" + temp_K_tmp_str + " }";
break;
    case DHTLIB_ERROR_CHECKSUM: 
jsonData = "{'state':'Checksum error'}"; 
break;
    case DHTLIB_ERROR_TIMEOUT: 
jsonData = "{'state':'Time out error'}"; 
break;
    default: 
jsonData = "{'state':'Unknown error'}"; 
break;
  }
  
  
  Serial.println(jsonData);

  delay(2000);
}
//
// END OF FILE
//

Most of what i changed was at the end of this script, I removed all the printed output that i didn't want, changed the way it handled errors and most importantly changed the output to a structure that resembled JSON.
I then attempted to read the data again and I was getting to something that was usable, so i moved over to writing a simple python script to read the data in, this python script needs to read in the data from the Arduino, then add a timestamp to the JSON, the timestamp is necessary for Splunk so it knows when the event occurred, and then output the new JSON.
The script i came up with is below, I am not completely happy with it, it loops through 3 iterations from the Arduino as i was finding that i was getting some duff characters on the first read, I plan to change this to take the first valid response based on what i expect to see,  but this script works for now.

import serial
import json
from datetime import datetime

port = serial.Serial("/dev/ttyUSB0", baudrate=115200, timeout=3.0)

for x in range(0,2):
    tmp_json = port.readline()

tmp_event_timestamp = datetime.now()
event_timestamp = str(tmp_event_timestamp)

try:
    jsonData = json.loads(tmp_json)
except:
    tmp_jsonData = '{"error":"unable to read a json string from arduino"}'
    jsonData = json.loads(tmp_jsonData)

jsonData["et"] = event_timestamp
print json.dumps(jsonData)

When this python is executed manually i get this as the output.

Now that i have some output I can get it into Splunk.  Splunk ships with it's own version of Python, and i am using the serial library which is not in that version, so i will need to get my script to use the operating system version of python, there are lots of ways that people do it using Splunk, but the simplest is just to pop it's invocation into a bash wrapper script, it's a little dirty, but works fine, the bash script i am using isthis.

#!/bin/bash
cd $SPLUNK_HOME/etc/apps/TA-beehive_sensors/bin
python dht11-internal.py

I have already deployed the Splunk Universal Forwarder for ARM onto my Raspberry Pi and I plan to put all of my sensor collection scripts into a new "Technology Add-on" app, in Splunk terms an app is just a bunch of config or scripts, they coined the term before the mobile boom of apps and app stores.
The Universal Forwarder will send all of the sensor telemetry to my existing Splunk instance.
To create my new TA, i have navigated to /opt/splunkforwarder/etc/apps and create a new directory called "TA-beehive_sensors", all of my config specific to this project will be placed in this folder, firstly i am going to put my first python script and the wrapper into a folder called bin.

The wrapper script will need to have it's execute bit set and your Splunk Universal Forwarder will need to be running as root due to the need to read the COM port.
Now that i have my scripts in place i just need to tell Splunk to do something with them, the file that I need to edit for this is the inputs.conf and that is located in the default or local folder within my "Technology Add-on"

The content of our inputs.conf will be:

[script://./bin/dht11-internal.sh]
interval = 60
sourcetype = temp_humidity
source = temp_humidity

This is a fairly simple input in splunk terms, Firstly we define that we want to collect data from a scripted input, and the path where we have deposited our wrapper script.
We then say how often we want to execute the script, in this case 60 means 60 seconds.
Then we have the sourcetype and source that we want to assign to the data we collect in Splunk, these will be used in the Splunk searches later, it makes data alot easier to manipulate in Splunk if you set these.
Now we will restart Splunk on the Raspberry Pi to apply our new configuration.
Now we move over to The web interface of my Splunk instance to see what we receive.

I have used the most simple search to see what i am getting, all i am doing here is calling all data for my sourcetype of temp_humidity for the last 60 minutes.  Success! We can see some data in our Splunk instance for my first sensor prototype.
I noticed that i was getting 2 distinct messages, these were the success and Error messages from our Arduino C code and Raspberry Pi Python, with the timestamp added by the Python script.  The errors I will investigate later, I suspect they are related to my ropey scripting, and i will clean them up later.
The error is caused then my Python script fails to read from the Arduino correctly.
Now that i have some simple data coming in from my sensors, even though this sensor was sat on my desk and not in a bee hive, I couldn't resist creating some simple charts in Splunk, to show off my success, after all a picture speaks a thousand words.  So 5 minutes later I had this:

The Top 2 are the most recent readings of humidity and Temperature in °C and the graph at the bottom is the temperature(Yellow Line) and humidity(Blue Line) for the last 24 hours on a time chart.
One final thing, while I was reading I came across two things that instantly presented themselves as perfect for this project, the first was a DHT11 sensor that had already been mounted onto a small circuit with the resistor in place and took me from a bundle of wires on a breadboard, to this:

The second was a different version of an Arduino called an Arduino Nano, this is much smaller, and if I had to add to the complexity of the project, I wanted to keep it small.
These 2 items arrived today, I have just assembled them and have now arrived at quite a clean solution for the first sensor in my project, the refined setup now looks like this.

So after nearly a week I have gone from nothing to having a working prototype of the temperature and humidity sensor that will be going into the hive(s).  I have gone through lots of iterations and learnt lots, I have also come to the realisation that this project is going to be a lot harder than i thought.
Have a good evening and I hope I haven't overloaded you with this post, more will follow over the coming days, weeks and months.