Home Automation using DASH7

Posted on January 4, 2013 by Fritz

The goal of the project was to implement a basic Home Automation using DASH7. We had limited resources, worked in a group with 7 people and had only 8 days to finish the project.

The goal of the project

The project overview

The idea is illustrated in the following image. A server is used and utilises a self-learning algorithm that learns preferences of the user. The user has a smart phone and can adjust the environment, these adjustments are registered on the server and used in the self-learning algorithm.

So a little example, I walk in the room and the light turns on, I find it to bright so change it to 70% from the original 100%. I do this every day for 3 days and on day 4 the light switches on, but instead of the 100% illumination, it is only on for 70%. The system learned my preference and applied it.

The smart phone communicates with the server using WiFi and the server must communicate with the application (light bulb, oven, water heater lamp, light strip …) using DASH7 technology.

We were free to choice the applications and the mobile application (dedicated, web-based universal or hybrid). There were only 8 days left and we were allowed to take short cuts (aka dirty code and tricks) to get a result.

DASH7 Technology

Everyone has an idea how a smart phone application might look. But the DASH7 part is a big unknown. There are developer kits from WizziLab for rather a hefty price (€100). A PhD student at my University made a smaller tag that only costs around €10, or even less.

: The wizzibase that expands the pins on the wizzimote

: The wizzimote without an antenna

Combine those two items and you can program the DASH7 tags. Right now the DASH7 Alliance is building a complete OSI stack for DASH7 and writing the code with the help of researchers/PhD students. They already finished the Physical and Data-Link Layer, we programmed using this stack on the Data-Link Layer.

This also means that we had to program embedded devices, luckily we could use C code and the library from the DASH7 Alliance include most of the important commands to send receive and handle interrupts. Quite cool using cutting-edge technology.

Task Assignment

So we (me and 6 colleagues) started to draft out a general concept/flowchart so we could divide this project in subsections. Every subsection is developed by one person who is responsible. We came up with the following flowchart.

Our basic flowchart, every colour represents a different person.

My job was to program the DASH7 tags and make sure the applications worked. Another student is on the same subsection as me but his task was to wire electronics on the GPIO pins of the DASH7 tags. I’ll only talk about my part because I don’t really know all the details.

DASH7 Application

We picked the following application. We have a house with 2 rooms, a living room with a TV and a workroom with a desk and a computer. When the users walks to the TV it should automatically play a video, if he walks away the video should pause. When the user enters the workroom the laptop should automatically open certain applications. Whenever the user leaves or enters the room the light should switch on or off accordingly.

The DASH7 application with two rooms

A couple of things come together here. Firstly, we have an approximation application, meaning that if a user comes near a tag something should happen. Secondly, we defined three types of DASH7 tags, a server tag, application tag and mobile tag. Finally, we interact with 230v light bulbs using the 3.3v GPIO from the DASH7 tag.

Communications in the DASH7 application

A little note, I made this communication protocol because its easy dirty and the way to go for our little application. In no way this is useful if more than 3 tags are used and way to much overhead comes in the mind. Because of the time constrains I picked this easy, dirty solution.

Commands

Every command/message is send using the following protocol that I also used in my ArduinoPi code with one big difference every command is send in one byte (0×00-0xFF). So small recap, every command starts with @ and ends with :, the first number indicates the command and the second the value. I also included the comma between the command and value but it’s not really needed since every value is only one byte. The reason was that later on I could always change it back to dynamically allocate the command and value.

Communications

So the communication is based on the following image (yes I love images).

Communications in the DASH7 application

Dotted lines indicated wireless communication and solid lines include wired communication. It’s quite simple, every command has its own value followed by the value (if any). The mobile tag always sends out a one every second, the user wears this tag (can be a watch or small keychain).

The application tag receives a one, pulls out the dbm value and sends it back as command 2 to the server tag. The server tag sends the dbm to the server over a serial connection and the server decides if the user is close enough to switch the light. Note that one short-cut is included, the application tag will also send the dbm value to the computer who will start the video if the user is close enough, normally the server should decide this (for various reasons we didn’t include this).

The server tag also receives a one and sends his dbm value to the server, which decides if the user is close enough to switch the lamp or start-up the applications on the computer.

Other communication is also included because we had to read out a sensor (light sensor), I used the same way of communication for the two light sensors.

Result

The following video demonstrates our project. We failed to include the smart phone (HTML5 based jQuery mobile application) because we were unable to connect it with the server/DASH7 tags. We used a WCF Service Library and apparently it is unable to start, talk or do anything when it receives commands from the smart phone. We only figured this out on the last day and try to convert it to a WCF Service but we had no more time and decided to remove it, shame really.

Conclusion

It was a fun project and I’m pleased with the results. We already had to present this project and the promoters/teachers were quite pleased but had some comments, mainly about the extendibility of our project.

Augmented Reality & Pattern Recognition Sources

Posted on December 27, 2012 by Fritz

I think this title is bigger then the content of this post, anyway after some request here is the source code of the augmented reality projected and the pattern recognition project I (and my school colleagues) created for our Vision course.

Augmented Reality

First you should host the application using XAMPP (Windows) or Apache and you also should use the latest chrome browser. I’ve never tested this on Firefox (don’t even bother trying to use IE or Safari). If you have a working apache/xampp installation then host the applications by copying all the files in the following archive in your www or htdocs folder.

Now surf to localhost and accept the use of the webcam. Normally you should see that your browser detects and uses your webcam. If this fails then try another webcam. In the archive I also included a word file with the markers you should print before using the application.

Pattern Recognition

Before you can use the following code you should make sure you installed OpenCV, libconfig, ARToolKit and ARToolKitPlus. I don’t plan on giving out my complete Visual Studio project (the project is originally from somebody else), but my basic commands and the most important function (classify) is included. This ensures that lazy-copy-cats will fail and people looking for information will succeed.

// The gaussian distrubution of the features calculated
float cLike(float val, int feature) {
	return (1/sqrt(2*M_PI*(pow(circle_parm[feature][1], 2))))*exp((pow(val-circle_parm[feature][0], 2))/(-2*pow(circle_parm[feature][1], 2)));
}

float nLike(float val, int feature) {
	return (1/sqrt(2*M_PI*(pow(noise_parm[feature][1], 2))))*exp((pow(val-noise_parm[feature][0], 2))/(-2*pow(noise_parm[feature][1], 2)));
}

float aLike(float val, int feature) {
	return (1/sqrt(2*M_PI*(pow(animal_parm[feature][1], 2))))*exp((pow(val-animal_parm[feature][0], 2))/(-2*pow(animal_parm[feature][1], 2)));
}

// This is the main classify class, it will threshold the marker, add some transformations
// and then also classify it
void classify(ARToolKitPlus::ARMarkerInfo* markerinfo, IplImage* img, int i) {
	// Only continue if area is big enough
	if(markerinfo->area < 1000) return;
	// First we must clear all temp data, just to prevent memory access issues
	cvZero(contMarkImage);
	cvZero(minCircleImage);
	cvZero(filledImage);
	cvZero(maskImage);
	cvZero(trainingset);

	// Warp the marker so that we obtain a square (undo the perspective transformation)
	// The resulting marker image can be used for classification
	IplImage* detectMarkImage = unWarp(img, markerinfo);
	cvSetImageROI(detectMarkImage, cvRect(markerinfo->pos[0]-50, markerinfo->pos[1]-50, markPixel, markPixel));
	cvShowImage("Detected Marker", detectMarkImage);
	cvResetImageROI(detectMarkImage);
	//Threshold the marker. Convenient for border detection
	IplImage* thresMarkImage = threshold(detectMarkImage, markerinfo);
	cvShowImage("Thresholded Marker", thresMarkImage);
	// return if the dimensions aren't correct, happens alot => cause of random crashes
	if(thresMarkImage->width != thresMarkImage->height) return;
	// Now we can copy everything
	cvCopy(thresMarkImage, trainingset);

	// Lets calculate a histogram and count the black and white pixels
	cvCalcHist(&thresMarkImage, histogram);
	float black = (float)*cvGetHistValue_1D(histogram, 0);
	float white = (float)*cvGetHistValue_1D(histogram, 1);
	float whiteBlackRatio = white / black;

	// There must be some white pixels, but markers arent more then half white, so we can check that
	if(whiteBlackRatio < 0.05 && whiteBlackRatio > 1) return;

	// Create a cross structur element 4x4 with a center of 2, 2
	IplConvKernel* crossStruct = cvCreateStructuringElementEx(4, 4, 2, 2, MORPH_CROSS);

	// Apply a closing and do it 3 times with the cross structure
	cvMorphologyEx(thresMarkImage, thresMarkImage, thresMarkImage, crossStruct, MORPH_CLOSE, 3);

	// Now expand the image with a border, attempt to remove the white small lines from the marker
	cvCopyMakeBorder(thresMarkImage, maskImage, cvPoint(border,border), 0);
	// Copy the image
	cvCopy(maskImage, filledImage);
	// Now apply the floodFill, flip it and combine the image to fill the holes
	cvFloodFill(maskImage, cvPoint(0,0), cvScalar(255));
	cvSubRS(maskImage, cvScalar(255), maskImage);
	cvAdd(maskImage, filledImage, filledImage);
	cvShowImage("Filled Marker", filledImage);

	// Some values for the parameters
	CvSeq* contour = 0;
	float areaTot = 0.0, lengthTot = 0.0, metric_cTot = 0.0, count = 0;
	float area = 0.0, areaMax = 0.0, metric_c = 0.0, metric_s = 0.0, metric_sTot = 0.0, length = 0.0, lengthMax = 0.0;

	cvZero(contMarkImage);
	CvMemStorage* storage = cvCreateMemStorage(0);
	// Find contours
	cvFindContours(filledImage, storage, &contour, sizeof(CvContour), CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE);

	// Contour total doesn't work at all, strange, lets count it ourself
	for(; contour != 0; contour = contour->h_next) {
		// Also draw contours (so pritty)
		cvDrawContours(contMarkImage, contour, cvScalar(255), cvScalar(255), 0, 0, 8);
		// Calculate all the different features
		area = fabs(cvContourArea(contour, CV_WHOLE_SEQ));
		length = fabs(cvArcLength(contour, CV_WHOLE_SEQ));
		areaTot += area;
		lengthTot += length;

		// make sure we fetch the biggest area and length
		if(area > areaMax) areaMax = area;
		if(length > lengthMax) lengthMax = length;
		// Also count the amount of contours we have
		count++;
	}
	// If the total area is bigger then 8000 we clearly don't have a marker
	if(areaTot > 8000) return;
	cvShowImage("Contour Function", contMarkImage);
	// The values will hold the probabilities for every class
	float cFeatures = 0.0, nFeatures = 0.0, aFeatures = 0.0;
	// Only continue if the values are above zero
	if(areaTot > 0 && lengthTot > 0) {
		// Calculate the circularity and squareness
		metric_c = areaMax * 4 * M_PI / pow(lengthMax, 2);
		metric_s = areaMax * 16 / pow(lengthMax, 2);
		metric_cTot = areaTot * 4 * M_PI / pow(lengthTot, 2);
		metric_sTot = areaTot * 16 / pow(lengthTot, 2);

		if(!trainingset_bool) {
			// Naive bayesiaanse classifier, using the blob count, circularity and black and white ratio
			cFeatures = cLike(metric_cTot, 0) * cLike(count, 1) * cLike(whiteBlackRatio, 2);
			nFeatures = nLike(metric_cTot, 0) * nLike(count, 1) * nLike(whiteBlackRatio, 2);
			aFeatures = aLike(metric_cTot, 0) * aLike(count, 1) * aLike(whiteBlackRatio, 2);

			// Now check for the highest probability, and then save the class name in the type variable
			if(cFeatures > nFeatures && cFeatures > aFeatures) {
				sprintf(type, "Circle");
			} else if(nFeatures > cFeatures && nFeatures > aFeatures) {
				sprintf(type, "Noise");
			} else {
				sprintf(type, "Animal");
			}
		}
	}

	// setup the OpenGL transformation
	updateTransformationMatrix(markerinfo);
	glMatrixMode(GL_MODELVIEW);
	glLoadMatrixf(gl_para);
	// Switch the color depending on the number of markers in screen
	glColor3f(color[i%5][0], color[i%5][1], color[i%5][2]);
	glLineWidth(1.0f);
	glBegin(GL_LINE_LOOP);
	glVertex2f(-40.0f, 40.0f);
	glVertex2f(40.0f, 40.0f);
	glVertex2f(40.0f, -40.0f);
	glVertex2f(-40.0f, -40.0f);
	glEnd();
	// Now print the text on the screen
	sprintf(type_string, "Type: %s", type);
	renderString(markerinfo->pos[0]-40, markerinfo->pos[1]+60, GLUT_BITMAP_HELVETICA_18, type_string);

	// Check the configuration, if we have to save data for feature selection or confusion matrix
	if(trainingset_bool) {
		float data[] = {count, whiteBlackRatio, markerinfo->area, lengthTot, areaTot, metric_cTot, metric_sTot, areaMax, lengthMax, metric_c, metric_s};
		int data_size = (sizeof data)/sizeof(data[0]);
		saveData(trainingset, data, data_size);
	} else if(predictclass_bool) {
		float data[] = {count, whiteBlackRatio, markerinfo->area, lengthTot, areaTot, metric_cTot};
		int data_size = (sizeof data)/sizeof(data[0]);
		saveData(trainingset, data, data_size, type);
	}

	cvClearMemStorage(storage);
	cvReleaseMemStorage(&storage);
	cvReleaseStructuringElement(&crossStruct);

}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

// The gaussian distrubution of the features calculated

float cLike(float val, int feature) {

return (1/sqrt(2*M_PI*(pow(circle_parm[feature][1], 2))))*exp((pow(val-circle_parm[feature][0], 2))/(-2*pow(circle_parm[feature][1], 2)));

}

float nLike(float val, int feature) {

return (1/sqrt(2*M_PI*(pow(noise_parm[feature][1], 2))))*exp((pow(val-noise_parm[feature][0], 2))/(-2*pow(noise_parm[feature][1], 2)));

}

float aLike(float val, int feature) {

return (1/sqrt(2*M_PI*(pow(animal_parm[feature][1], 2))))*exp((pow(val-animal_parm[feature][0], 2))/(-2*pow(animal_parm[feature][1], 2)));

}

// This is the main classify class, it will threshold the marker, add some transformations

// and then also classify it

void classify(ARToolKitPlus::ARMarkerInfo* markerinfo, IplImage* img, int i) {

// Only continue if area is big enough

if(markerinfo->area < 1000) return;

// First we must clear all temp data, just to prevent memory access issues

cvZero(contMarkImage);

cvZero(minCircleImage);

cvZero(filledImage);

cvZero(maskImage);

cvZero(trainingset);

// Warp the marker so that we obtain a square (undo the perspective transformation)

// The resulting marker image can be used for classification

IplImage* detectMarkImage = unWarp(img, markerinfo);

cvSetImageROI(detectMarkImage, cvRect(markerinfo->pos[0]-50, markerinfo->pos[1]-50, markPixel, markPixel));

cvShowImage("Detected Marker", detectMarkImage);

cvResetImageROI(detectMarkImage);

//Threshold the marker. Convenient for border detection

IplImage* thresMarkImage = threshold(detectMarkImage, markerinfo);

cvShowImage("Thresholded Marker", thresMarkImage);

// return if the dimensions aren't correct, happens alot => cause of random crashes

if(thresMarkImage->width != thresMarkImage->height) return;

// Now we can copy everything

cvCopy(thresMarkImage, trainingset);

// Lets calculate a histogram and count the black and white pixels

cvCalcHist(&thresMarkImage, histogram);

float black = (float)*cvGetHistValue_1D(histogram, 0);

float white = (float)*cvGetHistValue_1D(histogram, 1);

float whiteBlackRatio = white / black;

// There must be some white pixels, but markers arent more then half white, so we can check that

if(whiteBlackRatio < 0.05 && whiteBlackRatio > 1) return;

// Create a cross structur element 4x4 with a center of 2, 2

IplConvKernel* crossStruct = cvCreateStructuringElementEx(4, 4, 2, 2, MORPH_CROSS);

// Apply a closing and do it 3 times with the cross structure

cvMorphologyEx(thresMarkImage, thresMarkImage, thresMarkImage, crossStruct, MORPH_CLOSE, 3);

// Now expand the image with a border, attempt to remove the white small lines from the marker

cvCopyMakeBorder(thresMarkImage, maskImage, cvPoint(border,border), 0);

// Copy the image

cvCopy(maskImage, filledImage);

// Now apply the floodFill, flip it and combine the image to fill the holes

cvFloodFill(maskImage, cvPoint(0,0), cvScalar(255));

cvSubRS(maskImage, cvScalar(255), maskImage);

cvAdd(maskImage, filledImage, filledImage);

cvShowImage("Filled Marker", filledImage);

// Some values for the parameters

CvSeq* contour = 0;

float areaTot = 0.0, lengthTot = 0.0, metric_cTot = 0.0, count = 0;

float area = 0.0, areaMax = 0.0, metric_c = 0.0, metric_s = 0.0, metric_sTot = 0.0, length = 0.0, lengthMax = 0.0;

cvZero(contMarkImage);

CvMemStorage* storage = cvCreateMemStorage(0);

// Find contours

cvFindContours(filledImage, storage, &contour, sizeof(CvContour), CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE);

// Contour total doesn't work at all, strange, lets count it ourself

for(; contour != 0; contour = contour->h_next) {

// Also draw contours (so pritty)

cvDrawContours(contMarkImage, contour, cvScalar(255), cvScalar(255), 0, 0, 8);

// Calculate all the different features

area = fabs(cvContourArea(contour, CV_WHOLE_SEQ));

length = fabs(cvArcLength(contour, CV_WHOLE_SEQ));

areaTot += area;

lengthTot += length;

// make sure we fetch the biggest area and length

if(area > areaMax) areaMax = area;

if(length > lengthMax) lengthMax = length;

// Also count the amount of contours we have

count++;

}

// If the total area is bigger then 8000 we clearly don't have a marker

if(areaTot > 8000) return;

cvShowImage("Contour Function", contMarkImage);

// The values will hold the probabilities for every class

float cFeatures = 0.0, nFeatures = 0.0, aFeatures = 0.0;

// Only continue if the values are above zero

if(areaTot > 0 && lengthTot > 0) {

// Calculate the circularity and squareness

metric_c = areaMax * 4 * M_PI / pow(lengthMax, 2);

metric_s = areaMax * 16 / pow(lengthMax, 2);

metric_cTot = areaTot * 4 * M_PI / pow(lengthTot, 2);

metric_sTot = areaTot * 16 / pow(lengthTot, 2);

if(!trainingset_bool) {

// Naive bayesiaanse classifier, using the blob count, circularity and black and white ratio

cFeatures = cLike(metric_cTot, 0) * cLike(count, 1) * cLike(whiteBlackRatio, 2);

nFeatures = nLike(metric_cTot, 0) * nLike(count, 1) * nLike(whiteBlackRatio, 2);

aFeatures = aLike(metric_cTot, 0) * aLike(count, 1) * aLike(whiteBlackRatio, 2);

// Now check for the highest probability, and then save the class name in the type variable

if(cFeatures > nFeatures && cFeatures > aFeatures) {

sprintf(type, "Circle");

} else if(nFeatures > cFeatures && nFeatures > aFeatures) {

sprintf(type, "Noise");

} else {

sprintf(type, "Animal");

}

// setup the OpenGL transformation

updateTransformationMatrix(markerinfo);

glMatrixMode(GL_MODELVIEW);

glLoadMatrixf(gl_para);

// Switch the color depending on the number of markers in screen

glColor3f(color[i%5][0], color[i%5][1], color[i%5][2]);

glLineWidth(1.0f);

glBegin(GL_LINE_LOOP);

glVertex2f(-40.0f, 40.0f);

glVertex2f(40.0f, 40.0f);

glVertex2f(40.0f, -40.0f);

glVertex2f(-40.0f, -40.0f);

glEnd();

// Now print the text on the screen

sprintf(type_string, "Type: %s", type);

renderString(markerinfo->pos[0]-40, markerinfo->pos[1]+60, GLUT_BITMAP_HELVETICA_18, type_string);

// Check the configuration, if we have to save data for feature selection or confusion matrix

if(trainingset_bool) {

float data[] = {count, whiteBlackRatio, markerinfo->area, lengthTot, areaTot, metric_cTot, metric_sTot, areaMax, lengthMax, metric_c, metric_s};

int data_size = (sizeof data)/sizeof(data[0]);

saveData(trainingset, data, data_size);

} else if(predictclass_bool) {

float data[] = {count, whiteBlackRatio, markerinfo->area, lengthTot, areaTot, metric_cTot};

int data_size = (sizeof data)/sizeof(data[0]);

saveData(trainingset, data, data_size, type);

}

cvClearMemStorage(storage);

cvReleaseMemStorage(&storage);

cvReleaseStructuringElement(&crossStruct);

}

Web-based Augmented Reality 3D car Encyclopedia

Posted on October 19, 2012 by Fritz

For school we (me and 3 of my fellow students) had to create “something with augmented reality and markers”, love the vague object description. We already had some experience with loading 3D models in openGL and decided to try a new approach.

Web-based Augmented Reality using Three.js

Last time I talked about ARToolkit for detecting markers in C++. This library has been ported to JAVA (called NyARToolkit) and this JAVA version has been ported to Action Script (called FLARToolkit) and finally this action script version has been ported to JavaScript (called JSARToolkit).

Three.js is the javascript “helper” library to draw webGL objects in a canvas DOM element. JSARToolkit and Three.js are combined to create a Web-based Augmented Reality application by Jerome Etienne (thanks for the code).

Our project starts from Jerome’s code and we decided to add 3D models and create a 3D car encyclopedia.

The setup for web-based augmented reality

The following image displays our setup, we are using marker number 0-5.

The user can switch between cars, using the lower 4 markers, the middle marker will show a rotating 3D model and the marker on the top will show text information about the car (from Wikipedia)

How does it work?

Explaining every part of Three.js is a bit overkill so I’ll keep it to the basics of our project. There are 3 callbacks when a marker is visible in JSARToolkit: onCreate, onDelete and onUpdate.

onCreate: When a new marker is detected then this function will be called. We add the marker to an array of current visible markers. Depending on the marker number we draw the correct model. For example marker 1-4 are used to display the right car icon, marker 5 is for the Wikipedia text and marker 0 is used for the rotation 3D model.

onDelete: When a marker is removed from the screen we also remove it from our current-visible-markers array and also delete the marker object.

onUpdate: Every frame the marker is still in the image we update the matrix and also check if we have to change the models.

If the user covers one of the icon markers a new model will be loaded and the right information text is shown.

The result

Video says more then words. Note that the frame rate is lower because I had to capture the web browser, the actual result has a higher frame rate. ~~The project will follow soon on~~ ~~GitHub.~~ Update: more information here

Fritz-Hut

About Electronics and Informatics

Category Archives: School

Home Automation using DASH7

The goal of the project

DASH7 Technology

Task Assignment

DASH7 Application

Communications in the DASH7 application

Commands

Communications

Result

Conclusion

Augmented Reality & Pattern Recognition Sources

Augmented Reality

Pattern Recognition

Web-based Augmented Reality 3D car Encyclopedia

Web-based Augmented Reality using Three.js

The setup for web-based augmented reality

How does it work?

The result

The goal of the project

DASH7 Technology

Task Assignment

DASH7 Application

Communications in the DASH7 application

Commands

Communications

Result

Conclusion

Share this:

Augmented Reality

Pattern Recognition

Share this:

Web-based Augmented Reality using Three.js

The setup for web-based augmented reality

How does it work?

The result

Share this: