Development of the Internal Temperature and Humidity Sensor

Details

This is the sensor that will be placed inside the brood box of the hive and will be used to monitor the internal temperature and humidity of the hive.
This sensor will be mounted into the inside wall of the hive.
Under normal circumstances the bees maintain a constant temperature and humidity inside the brood box.  In a healthy beehive, the bees are really good at keeping this constant, so changes to these norms could be an indicator that something is wrong within the beehive.

This is going to be harder than i thought...

On Sunday I started with this circuit, I took the designs from this guide from Adafruit.  Given that this is a complete guide on how to make this work I thought it was going to be an easy starting point, boy was i wrong...

I followed the guide and built this sensor, it looks like a mess, but is only a prototype, so function over beauty it is...
On Sunday i spent several hours trying to read this sensor using the Raspberry Pi and Python with little success, i rebuilt this breadboard several times, replacing all the components incase something was faulty and still i could not get any readings.  I did make it a little harder than i needed to as i was stubborn and wanted to learn how to read from this sensor myself rather than using the Adafruit library in the guide, this did help me to learn a lot...
After several hours and much time reading through various articles and guides I came across a few mentions that the DHT11 sensor when using a Raspberry Pi and Python as the programming language can be problematic, people had had better success using C, but it was still not 100% accurate.  On further reading this appeared to be because the kernel in linux doesn't give enough processing time to Python or C in order for the data to be received from the sensor.  The sensor sends a blob of data when asked to provide a reading.
I happened to have an Arduino to hand so i connected it up to the sensor that i had already built to see if i got any better results from that, and I did.  So i now knew that my circuit was good and it was definitely something to do with the scheduling of the processes, this was a great relief as i then knew that my sensor and circuit worked, but i still had the issue of getting it to work with the Raspberry Pi.
I finally solved this problem when i found an article that people were using Arduino's and Raspberry Pi's in combination with great success.  Basically they used the Ardunio specifically to collect data from the sensor and then the Raspberry Pi would read the data from the Arduinos USB port.  Here us a great article explaining it all.
I then built a simpler circuit for the Arduino, and connected the sensor, the Arduino and the Raspberry pi together, and i ended up with this.

I started off with a simple C program on the Arduino that collected the sensor information and outputted it to the screen, I don't know C so i started off with a piece of code that i downloaded from here and uploaded it to the Arduino.
I then used my raspberry pi to read the data from the Arduino, by treating it as a COM port.
This showed me that i was able to receive data to the Raspberry Pi from the Arduino and this was a good achievement, but the output was not what i wanted, i decided i was going to make the Arduino output in a format called JSON, this is a nice easy to read structured data format, so i re-wrote the C program that i got from UU gear and turned it into this:
// 
//   FILE:  dht11_test1.pde
// PURPOSE: DHT11 library test sketch for Arduino
//

//Celsius to Fahrenheit conversion
double Fahrenheit(double celsius)
{
return 1.8 * celsius + 32;
}

// fast integer version with rounding
//int Celcius2Fahrenheit(int celcius)
//{
//  return (celsius * 18 + 5)/10 + 32;
//}


//Celsius to Kelvin conversion
double Kelvin(double celsius)
{
return celsius + 273.15;
}

// dewPoint function NOAA
// reference (1) : http://wahiduddin.net/calc/density_algorithms.htm
// reference (2) : http://www.colorado.edu/geography/weather_station/Geog_site/about.htm
//
double dewPoint(double celsius, double humidity)
{
// (1) Saturation Vapor Pressure = ESGG(T)
double RATIO = 373.15 / (273.15 + celsius);
double RHS = -7.90298 * (RATIO - 1);
RHS += 5.02808 * log10(RATIO);
RHS += -1.3816e-7 * (pow(10, (11.344 * (1 - 1/RATIO ))) - 1) ;
RHS += 8.1328e-3 * (pow(10, (-3.49149 * (RATIO - 1))) - 1) ;
RHS += log10(1013.246);

        // factor -3 is to adjust units - Vapor Pressure SVP * humidity
double VP = pow(10, RHS - 3) * humidity;

        // (2) DEWPOINT = F(Vapor Pressure)
double T = log(VP/0.61078);   // temp var
return (241.88 * T) / (17.558 - T);
}

// delta max = 0.6544 wrt dewPoint()
// 6.9 x faster than dewPoint()
// reference: http://en.wikipedia.org/wiki/Dew_point
double dewPointFast(double celsius, double humidity)
{
double a = 17.271;
double b = 237.7;
double temp = (a * celsius) / (b + celsius) + log(humidity*0.01);
double Td = (b * temp) / (a - temp);
return Td;
}


#include <dht11.h>

dht11 DHT11;

#define DHT11PIN 2

void setup()
{
  Serial.begin(115200);
}

void loop()
{

  int chk = DHT11.read(DHT11PIN);

  String jsonData;

  char humidity_tmp[10];
  dtostrf((float)DHT11.humidity,1,2,humidity_tmp);
  String humidity_tmp_str;
  humidity_tmp_str = String(humidity_tmp);
  char temp_C_tmp[10];
  dtostrf((float)DHT11.temperature,1,2,temp_C_tmp);
  String temp_C_tmp_str;
  temp_C_tmp_str = String(temp_C_tmp);
  char temp_F_tmp[10];
  dtostrf(Fahrenheit(DHT11.temperature),1,2,temp_F_tmp);
  String temp_F_tmp_str;
  temp_F_tmp_str = String(temp_F_tmp);
  char temp_K_tmp[10];
  dtostrf(Kelvin(DHT11.temperature),1,2,temp_K_tmp);
  String temp_K_tmp_str;
  temp_K_tmp_str = String(temp_K_tmp);

  //Serial.print("Read sensor: ");
  switch (chk)
  {
    case DHTLIB_OK: 
jsonData = "{ \"humidity\":" + humidity_tmp_str + ", \"temp_C\":" + temp_C_tmp_str + ", \"temp_F\":" + temp_F_tmp_str + ", \"temp_K\":" + temp_K_tmp_str + " }";
break;
    case DHTLIB_ERROR_CHECKSUM: 
jsonData = "{'state':'Checksum error'}"; 
break;
    case DHTLIB_ERROR_TIMEOUT: 
jsonData = "{'state':'Time out error'}"; 
break;
    default: 
jsonData = "{'state':'Unknown error'}"; 
break;
  }
  
  
  Serial.println(jsonData);

  delay(2000);
}
//
// END OF FILE
//

Most of what i changed was at the end of this script, I removed all the printed output that i didn't want, changed the way it handled errors and most importantly changed the output to a structure that resembled JSON.
I then attempted to read the data again and I was getting to something that was usable, so i moved over to writing a simple python script to read the data in, this python script needs to read in the data from the Arduino, then add a timestamp to the JSON, the timestamp is necessary for Splunk so it knows when the event occurred, and then output the new JSON.
The script i came up with is below, I am not completely happy with it, it loops through 3 iterations from the Arduino as i was finding that i was getting some duff characters on the first read, I plan to change this to take the first valid response based on what i expect to see,  but this script works for now.

import serial
import json
from datetime import datetime

port = serial.Serial("/dev/ttyUSB0", baudrate=115200, timeout=3.0)

for x in range(0,2):
    tmp_json = port.readline()

tmp_event_timestamp = datetime.now()
event_timestamp = str(tmp_event_timestamp)

try:
    jsonData = json.loads(tmp_json)
except:
    tmp_jsonData = '{"error":"unable to read a json string from arduino"}'
    jsonData = json.loads(tmp_jsonData)

jsonData["et"] = event_timestamp
print json.dumps(jsonData)

When this python is executed manually i get this as the output.

Now that i have some output I can get it into Splunk.  Splunk ships with it's own version of Python, and i am using the serial library which is not in that version, so i will need to get my script to use the operating system version of python, there are lots of ways that people do it using Splunk, but the simplest is just to pop it's invocation into a bash wrapper script, it's a little dirty, but works fine, the bash script i am using isthis.

#!/bin/bash
cd $SPLUNK_HOME/etc/apps/TA-beehive_sensors/bin
python dht11-internal.py

I have already deployed the Splunk Universal Forwarder for ARM onto my Raspberry Pi and I plan to put all of my sensor collection scripts into a new "Technology Add-on" app, in Splunk terms an app is just a bunch of config or scripts, they coined the term before the mobile boom of apps and app stores.
The Universal Forwarder will send all of the sensor telemetry to my existing Splunk instance.
To create my new TA, i have navigated to /opt/splunkforwarder/etc/apps and create a new directory called "TA-beehive_sensors", all of my config specific to this project will be placed in this folder, firstly i am going to put my first python script and the wrapper into a folder called bin.

The wrapper script will need to have it's execute bit set and your Splunk Universal Forwarder will need to be running as root due to the need to read the COM port.
Now that i have my scripts in place i just need to tell Splunk to do something with them, the file that I need to edit for this is the inputs.conf and that is located in the default or local folder within my "Technology Add-on"

The content of our inputs.conf will be:

[script://./bin/dht11-internal.sh]
interval = 60
sourcetype = temp_humidity
source = temp_humidity

This is a fairly simple input in splunk terms, Firstly we define that we want to collect data from a scripted input, and the path where we have deposited our wrapper script.
We then say how often we want to execute the script, in this case 60 means 60 seconds.
Then we have the sourcetype and source that we want to assign to the data we collect in Splunk, these will be used in the Splunk searches later, it makes data alot easier to manipulate in Splunk if you set these.
Now we will restart Splunk on the Raspberry Pi to apply our new configuration.
Now we move over to The web interface of my Splunk instance to see what we receive.

I have used the most simple search to see what i am getting, all i am doing here is calling all data for my sourcetype of temp_humidity for the last 60 minutes.  Success! We can see some data in our Splunk instance for my first sensor prototype.
I noticed that i was getting 2 distinct messages, these were the success and Error messages from our Arduino C code and Raspberry Pi Python, with the timestamp added by the Python script.  The errors I will investigate later, I suspect they are related to my ropey scripting, and i will clean them up later.
The error is caused then my Python script fails to read from the Arduino correctly.
Now that i have some simple data coming in from my sensors, even though this sensor was sat on my desk and not in a bee hive, I couldn't resist creating some simple charts in Splunk, to show off my success, after all a picture speaks a thousand words.  So 5 minutes later I had this:

The Top 2 are the most recent readings of humidity and Temperature in °C and the graph at the bottom is the temperature(Yellow Line) and humidity(Blue Line) for the last 24 hours on a time chart.
One final thing, while I was reading I came across two things that instantly presented themselves as perfect for this project, the first was a DHT11 sensor that had already been mounted onto a small circuit with the resistor in place and took me from a bundle of wires on a breadboard, to this:

The second was a different version of an Arduino called an Arduino Nano, this is much smaller, and if I had to add to the complexity of the project, I wanted to keep it small.
These 2 items arrived today, I have just assembled them and have now arrived at quite a clean solution for the first sensor in my project, the refined setup now looks like this.

So after nearly a week I have gone from nothing to having a working prototype of the temperature and humidity sensor that will be going into the hive(s).  I have gone through lots of iterations and learnt lots, I have also come to the realisation that this project is going to be a lot harder than i thought.
Have a good evening and I hope I haven't overloaded you with this post, more will follow over the coming days, weeks and months.

The Parts of a Beehive

Introduction

So far i have talked about the technical details on what i am trying to achieve and how.  I thought that before i start to merge the technical with the beehive that we should take a moment to talk through the various parts of a beehive.

There are many types of beehive available to the beekeeper, when I was kindly given my first colony of bees by Peter Dight I was given a colony that was in a national hive, ever since then i have used national hives, they are very popular in the UK and are easy to work with when you open them up, they basically look like a bunch of square boxes stacked on top of each other.  

Here is one of my hives to give you an idea.

This picture was taken shortly after catching a swarm of bees that decided they wanted to join in my friends wedding festivities this year.

The Hive

National Hives are modular, they are very flexible and simple to assemble, which based on my experience seems to be why people like them.

All my hives are made from cedar, i like using cedar as it is long lasting and also resistant to lots of insect attacks, also when i am making my hives, the smell of cedar wood is lovely.

They are Typically made up of an entrance, a brood box, a queen excluder, 1 or more super's, a crown board and a roof.

All of the required parts simply stack on top of each other.

In the photo above of the small hive, there is an entrance, a brood box and a roof visible.

We'll walk through the parts that you may need to know now.

Entrance

The entrance is where all of our bees enter and leave the hive, this is where we will be mounting our bee counter, across the front of the gap that is visible in the photo above.

Frames

Frames are what goes in each of the boxes, you can normally get 12 frames in each box, there are typically 2 sizes of frames that people use, ones that fit the brood box and ones that fit the supers.

The frames are exactly that, a wooden frame that has a sheet of bees wax attached, this wax is called foundation.  

We put these into hives to give the bees something to build their honey comb against, if we just gave them empty boxes, the bees would be perfectly happy, but when it came to checking on them, or harvesting their honey it would be very difficult, as the bees would build whatever shapes they wanted to and it would be quite organic, the frames give structure that make our lives easier as beekeepers. 

Once the bees have drawn out the frames with comb (this is a fancy way of saying that they have made the little hexagonal cells that they deposit the honey into), they will then fill the cells up with the precious nectar that they collect.

Here is a picture i took a few years ago of the drawn out comb on a frame.

It still fascinates me that every cell looks identical, the cell walls are made of wax.

Brood Box

This is normally the deeper box that is located near the entrance to the hive, it is where the queen lives(and should be found).  She lays her eggs in the cells and her and her young are nursed by some of the bees within the hive.

Queen Excluder

The queen excluder is a divider that sits between the brood box and the super, it is normally metal or plastic and contains lots and lots of holes.

It is designed so that the holes are big enough for the worker bees to fit through, but small enough that the queen can't get through into the upper parts of the hive.  We don't really want the queen laying eggs, or have young larvae creating waste and detritus in the supers, we want to keep the honey in the supers clean(for us to steal).

Super

This is where the bees deposit nice clean honey for their reserves, so that they have plenty of food in stock.  This is the honey that we collect as rent for letting the bees live in our hives...

Back to the Sensors.

Most of our sensors will be concentrated around the bottom sections of the hive, specifically the entrance and the brood box.  With the scales sitting under the entrance section, the bee counter being attached to the entrance and the internal temperature and humidity sensor being mounted on the inside wall of the brood box.  We may add additional sensors as we make this journey, as of today though they are the planned sensors and mounting locations.

The Bee Counter - The Concept

What is the bee counter?

The bee counter is going to be the most complex sensors in this project.  The idea is that i will be using it to count bees in and out of the hive.  

The bee counter will sit in front of the entrance to the beehive and will be made up of 20 gaps, each gap will be small enough that only 1 bee can pass through at a time in 1 direction, i can then record that a bee was entering or leaving the hive using the sensors.  I have taken the idea from an link i found when searching through google while trying to work out how to count bees.

How will it work?

It will be made up of at least 40 phototransistors and these will be used in pairs.  In simple terms the phototransistor will register when something travels past the sensor and reflects the light emitted into a receiver.

The pairs will be placed in between two blocks allowing only a single bee to pass through at a time, the direction that the bee is travelling will be determined by the order in which the photo transistors are triggered.  There are some diagrams below that show this better, the black blocks represent the phototransistors.

Lets assume that the sensor on the outer edge is number 1 and the sensor on the inside edge is number 2, if the sensors are triggered 1 then 2 the bee is travelling into the hive, if it is 2 then 1 it is leaving the hive, well that's the theory anyway.

I plan to build these as 4 units, each with 5 gates, this will simplify the project, keeping it in manageable chunks, while also make it easier to diagnose and swap out faulty components, should the need occur.

My Initial Sketches

My Sketches Drawn up in Google Sketchup

Single Unit

4 Units Side by Side

The Next Steps For the Bee Counter.

Now that i have the idea and direction, I have ordered the relevant components, once they arrive I will start to create the circuits on a breadboard and hook them up to the raspberry pi so that i can see how they work.  I am not an expert with circuits, I remember playing with them at school and get the concept, but it's been a few years, so if I struggle i may call in favours from friends that are better at it that me.

Initially I plan to workout the bugs on a small scale using just 2 phototransistors, once i understand how to read the data from the sensors i will then build out 1 of the 4 units on the breadboard and then test that.  Finally once I am happy that what i have built works I will turn that circuit layout from the breadboard into a proper circuit board, I am hoping to use a piece of software I have been told about called Fritzing to make that easier, from what I have been told it allows me to draw out what i have built on the breadboard and will turn that into a proper circuit board diagram that i can then send off to have made properly.

Why have I chosen the Raspberry Pi over one of the many other single board computers that are available?

Project Requirements

• Cheap single board computer with low running costs and total cost of ownership. (TCO)
• Support for alot of digital inputs. (currently i have calculated i need about 50 separate inputs)
• Wireless communication to allow data to be sent to my Splunk instance in near realtime.
• Customisable and extensible, either using add on cards or USB.

Project Decision

There are many single board computers that are available, i looked at the Raspberry Pi, Arduino and Beaglebone Black, all cost under £40 and could all meet my requirements.

Arduino

My initial thoughts were to use an Arduino, they are quite popular at the moment as a cheap unit for receiving sensor data, and triggering relays and like, there is also an active community around the projects that you can do with these devices.

These are fairly basic boards with wireless and USB having to be provided via an additional add on cards, if i was using some form of static collection and upload via SD card or similar then they would of been perfect, but as i want a near realtime data feed for automated analysis, the cost of the required add on cards would of pushed the TCO options available to me.

Beagle Bone Black or Raspberry Pi

OR 

After i ruled out the  Arduino i was left with 2 options from my chosen top 3, the Beaglebone Black and the Raspberry Pi, they both offer me similar functionality.  They have USB port(s) which would give me the ability to use a cheap USB wifi adaptor to send data in near real time.   I am able to run linux on both of these and I am also able to run a Splunk Agent to send the collected data and also a Puppet agent for configuration management and enforcement(more on this later as it may seem unnecessary to some people).

Given that they are both very similar and would both be able to meet my requirements, why did i choose the Raspberry Pi.  Basically the Pi is slightly cheaper as an initial purchase and also has a more active community supporting it.  This means I would have a larger number of options available to me when it comes to decisions later on in the project, I will also have a much richer community that I can reach out to if i struggle to make something work.

How i plan to answer my questions using sensors?

Introduction

At this state these are just thoughts and theories on how i will answer the questions i have posed.

My questions

• Are my bees healthy?
• Is my colony strong/growing?
• Are my bees starving or struggling to find a food source?
• Have my bees swarmed?
• Is there any honey for me to harvest?

What sensors do i plan to use on each Beehive?

Combined Temperature and Humidity Sensor

I plan to install this into the inside of my brood box, set into the wall of my hive.  This will tell me the humidity and temperature inside the brood box.

Waterproof Temperature Sensor

This will go outside the hive so that i can measure the ambient temperature around the Beehive.

Half Bridge Resistance Sensors

These will be used to create a set of scales that the Beehive will permanently rest on, i plan to install them between hive stand and the open mesh floor of the hive.

Miniture Reflective Object Sensors

These will be used in pairs to allow me to count bees in and out of the hive entrance, these will create the largest set of data, and are also the most complicated part of what i plan to do, my initial estimates are that i will need  about 46 of these per hive, which will give me 23 slots for bees to pass through single file to get in and out of the hive.  I will go through the concept of the bee counter in a later post.

How the sensors will help me answer my questions?

Is my colony strong/growing?

My theory here is to use two of the sensors listed above this to measure how healthy my colony is, this will give me an indicator.  The sensors i plan to use are the weight of the hive, as my thoughts are that a hive that is getting heavier is a happy hive, as the bees are increasing in numbers and are actively collecting and storing food stores.  My second sensor is to use the bee counter made up of the reflective object sensors to count bees in and out of a hive, my theory here is that as a colony is getting stronger the average number of bees that leave and return to the hive on a daily basis will increase, I may also be able to measure the attrition rate of bees that leave the hive, but do not return.

Are my bees starving or struggling to find a food source?

This question came about after the funny summer that we have had this year, there seemed to be a gap between the summer flowers finishing and the autumn flowers becoming available to the bees, this meant that all of my hives suddenly started to use all of the reserves they had been collecting, which puts them at risk of starvation if they were to run out completely.  So i plan to use the scales and bee counter to answer this question again, if the number of bees entering and leaving hasn't massively changed over a given period, but the hive is getting lighter quickly, then they are probably eating their reserves, which is obviously not a great sign.

Have my bees swarmed?

This will be a simple check using the bee counter, if a massive number of bees leave the hive in a short space of time, and don't return, this could be an indication of a swarm event.  

Also i maybe able to learn swarm indicators from the behaviour of the Beehive in the days or hours leading up to the swarm that i can use in the future.

Is there any honey for me to harvest?

This will simply be based on the weight measurements from the scales.

Another question that i have been thinking about.

This is a question that i have thought about, and will be a nice to have answers to, but is secondary to my questions above.

Can i use the internal and external temperature readings for anything?

One possible scenario that i haven't had happen to me, but i know fellow beekeepers that have suffered, is woodpeckers, they tap a hole into the side of your hive and then cause havoc to your bees, often killing the colony completely and leave a nasty whole in your hive that will need repairing.  I think that i may be able to spot that this is/has happened using the temperature differential between the internal and external temperature, coupled with a sudden drop in temperature or change of humidity inside the Beehive, for those that don't know, bees are very good at controlling the internal temperature and humidity of a hive and under normal circumstances i expect this to be fairly constant.

Introduction To the Project

Who am I?

My name is Darren, I currently live in the UK and I work with in the interesting field of Big Data Analytics and predominantly work with a software product called Splunk.  In my spare time, among other things i also keep bees and currently have 10 beehives dotted around the area i live, mostly in friends and my own back gardens.
I got into beekeeping a few years ago, when i wanted to learn about the decline of bees and the effect that would have on society as we know it.  Rather than reading a book, i decided to learn by doing and got my first hive.

The Project

What I intend to do and Why?

The goal of this project was to marry my career and my hobby in an interesting way, i also wanted to learn about the health and strength of my colonies when i am not near the hives.  Also it is an interesting challenge for personal learning and development for my career, to make me better at what i do.
I am also going to attempt to write this blog to track my progress, let anyone that is interested know what i have done, so if they want they can repeat and improve on what i have done.  I am also going to attempt to make this blog understandable and hopefully interesting to both my beekeeping friends and people that work within my industry, at the end of the day we can all hopefully learn something new

Project Goals

The main "use case" that gave me the idea to start this project was that i wanted to know how healthy my colonies are without having to disturb them all the time, the questions I hope to answer as part of this project are:

• Are my bees healthy?
• Is my colony strong/growing?
• Are my bees starving or struggling to find a food source?
• Have my bees swarmed?
• Is there any honey for me to harvest?

I am sure that i will learn other things while i work on this project, but for these are my key indicators for success.
Once I am collecting data, all output from this project should be understandable by as many people as possible.

Personal Learning Goals

One of the reasons for thinking about a project like this was career development, i find it easier to learn when i have a real objective, rather than just reading books and knowledge articles.  One of the things i have been trying to improve is a programming language called python and i intend to use that to take readings from the sensors and feed that information to Splunk for analysis.  I am also going to be using a few other work related tools for the technical bit, which are an operating system called Linux and a configuration management tool called Puppet.  The hardware that i intend to use is a small computer that is being used in schools a lot now to teach programming called a Raspberry Pi.