Andddd We're Done!

With the final tweaks mentioned in previous posts, we were able to decode 24 of the 26 letters of the alphabet to some satisfactory degree of accuracy. Of course, some letters decode more robustly than others, and the letters O and V remain to be implemented. O is very similar to C and E in configuration, and V is very similar to R and U. With a couple more sensors, it would have been possible to decode these last two letters as well (and perhaps make some of the others more robust). However, we had maxed out the number of GPIO pins on the Firefly. With a change of platform, or additional pins, we would be able to decode the full alphabet.

The final product was demonstrated at the ESE350 Demo Day as well as an additional demo day for professors and other students. We had a great time showing our work and were glad that the final product was a success!

Also, find our super awesome video here:

Relaxing Constraints, Debouncing, and Error Checking

Even after implementing the buckets, we didn't get very great accuracy.  Often, some fingers would register in the wrong state, especially fingers that were supposed to be half-bent.  We realized that many sets of finger positions were not an ASL sign for a letter.  We decided to relax some constraints by saying that for each letter, fingers could be in multiple positions and still be decoded as that letter.  This greatly increased the accuracy of the system, and made it SO MUCH COOLER!

Another important thing that we added was a debouncing/error-checking algorithm.  For debouncing, we stored the last letter printed, and ensured that each letter could only be printed once per successful decode.  For the error-checking, we made an array of recently decoded letters.  When the past seven letters are the same, the letter is printed.  This helped to stop the printing of transition letters - letters that get decoded while the hand is moving from one letter to the other.

P.S. pictures, video, and final report coming soon!

Buttons and Buckets, my oh my!

So our ADC ranges were self-regulatory by the method of k-means, but we still had this problem of changing our bucket values too much when certain extreme letters were signed. We implemented a possible solution to this problem: have finite bucket ranges! For example, if the reaveraged bucket value exceeded the maximum bucket range (relative to the initial seed bucket value) for a given finger, then the bucket value would simply take on the value of the maximum bucket range. This way, extreme letters in the language do not pull the average buckets to values that cannot be reached with other gestures. This seemed to work much better, and certainly took care of our problem of precluding letters by evolving our buckets too much. Bucket, bucket, bucket, bucket. Phew, now that that's out of my system...

On to buttons! So we had been thoroughly avoiding the decoding of letters M, N, and T. As the image shows, they're almost the exact same hand configuration, with an ever so slight variation of which two fingers the thumb is in between. M has the thumb between the pinky and the ring finger; N between the ring finger and the middle finger, and T between the middle finger and index finger. The degree of thumb bending is not enough by itself to decode the letters. So we finally sucked it up and decided to add button sensors that would go on the pinky (M), the ring (N), and middle (T) fingers to tell the difference between them. The button was also used as an indication of U versus V later on, even though this is slightly unnatural for sign language. The addition of these sensors worked great for those five letters (M, N, T, U, V)! See the image below, taken after adding buttons to our glove. Definitely raises the cool factor by about 1000.

Decoding Letters

Once the glove was completely built, we could then reliably take note of ADC ranges for each finger. We found that a completely bent finger was different enough from a half-bent finger or from a completely unbent finger for us to differentiate between the three states. This gave us confidence that the letters would be separable for the most part.

Recall that the ADC circuit was built using a voltage divider circuit which had a resistor connected to Vdd and the flex sensor connected in series to ground. Since the sensor increases resistance with flex, bending a finger corresponds to higher ADC values. As a first step, we started to decode letters simply by imposing threshold values for each finger, and printing them to the terminal. For example, the letter 'A' (see image below) has a thumb unbent and the rest of the fingers bent. So, 'A' was decoded by an if statement that checked whether the thumb's flex sensor was below a certain threshold (unbent) and the other four fingers' flex sensors were each above a certain threshold (bent). This strategy worked fantastically ... for letter 'A'. Even 'B' worked pretty well. So yes, there was reason to celebrate a good week into the project. But we hit some problems with this method when trying to decode other letters, such as 'C'. As can be seen below, all of the fingers are in a half-bent position for 'C'. We found that it was much easier to detect a bent state or an unbent state because they are on the extremes of the ADC spectrum. Half-bent, however, is a very noisy state, and its ADC range can vary from day to day and even from trial to trial.

To gather some more information on exact nature of our problem, we attempted to decode a few more letters, and found that the half-bent state is going to be our challenge for the majority of letters in ASL. Perhaps we should try something a bit more adaptive than the absolute threshold method? Yes, you've guessed it; we're sending our glove to school!

Our next step was to employ some version of k-means as a learning algorithm. Perhaps in time, our glove will be able to do our 350 project for us! (I suppose that would be a project all by itself). Anyway, our method involved creating 3 "buckets" for each finger, representing the three possible states. These buckets were initialized to the minimum, maximum, and middle ADC values for each finger. As each ADC value comes in for a finger, it is compared with each of the three elements of the bucket. The bucket that the value is closest to gets averaged with the value so as to update the bucket. In this manner, the three "bent", "half-bent" and "unbent" states are automatically calibrated after a few averages.

This new algorithm seems to be running a lot smoother and more consistently. However, averaging sometimes pulls the bucket values to extremes. The program is able to decode a certain letter, but it can no longer do so after signing another letter since the buckets have changed. Or vice versa, one must pull the average down before it can detect a letter. As can be seen, there is some work to be done to play with the k-means implementation. We do believe this is a better way of decoding, however, than our original thresholds.

Unfortunately, our pinky broke a couple days ago! Sad =(. Sid and Roop have ordered a new sensor along with some extras so that we can resume our project right away. In the mean time, Bob was nice enough to lend us one of his sensors! Another problem that we faced was a sudden reduction in resistance range with our sensors. We think that heavy usage is the cause of the range reduction, and hope that this doesn't happen again with the new sensors. A possible option is to decrease the voltage divider resistance in expectation of the reduced range, so that our ADC range will not shrink. But imagine opening up that beautifully soldered voltage divider circuit. Ew.

Phew! Well there you have it. We'll keep you updated! Oh, by the way, wireless transmission has been implemented so I am no longer tethered to the computer!

Finished Glove, Demo, and Next Steps

We finished building the entire glove on Tuesday night.  This involved trimming the leads coming from the flex sensors, soldering all the grounds to one common node, attaching resistors and ADC leads, and connected the resistors to one Vdd node. This proved a little more difficult than expected, simply because of the small space we were working in.  In addition, we attempted to lead the wires to reinforce the extra stitching we did to keep the flex sensors in the optimal flexing area.  After some testing, we realized that the thumb and pinky in particular weren't really placed well.  To rectify this, we added extra stitches to keep them in place.  We also reinforced the other sensors to ensure a tight connection.

As a side note, every wire is color coded.  :)  Red = Vdd, Black = ground (of course), Blue = flex sensor connection, orange = ADC connection.


We demoed our glove on Wednesday to Professor Mangharam.  We showed him the finished glove, and the ADC values on the terminal.  We were able to show him the ranges of ADC values we were getting and the hopefully big enough margin between straight and bent.

Next, we plan on decoding a couple of letters, starting with the simple ones (A, for one).  We'd also like to get wireless transmission working.  We've been forced to do all computation on the node connected to the glove because the other node wasn't functioning.  We have since gotten more nodes, and hope to get this working so that the glove isn't tethered to the computer.

Making the Glove

The next step was to make the glove.  We decided to go with a thin fully cloth glove so that the flex sensors could be sewn on, and so that they would be tight against the hand.  We need as much sensitivity as possible.

Initially, we were planning on making thread holders for the flex sensors, similar to how staples are used to hold wires to walls.  However, this was really tedious and didn't provide a very tight grip for the sensors.  So, we thought for a bit.  What's the best way to mount flex sensors to a glove?

Thinking...

Thinking...

WOW.  So the flex sensors were very flimsy, particularly at the contact point.  They came reinforced everywhere except the very end (yes, how incredibly stupid).  Anyway, we decided to use electrical tape and wind it around the end to reinforce it.  THEN, we thought, "why not put electrical tape around the whole thing?!"  By leaving a flap around the flex sensor, we could sew the electrical tape to the glove using standard sewing methods, and it would be PERFECT.

So, voila!  Behold: THE GLOVE.  We still have some modifications to make (including cleaning up that frightening mass of wire), but this is the basic design.  The idea is to mount the Firefly on the back of the glove, and run wires around the glove as necessary.  We also have to solder resistors between the blue wires and Vdd to complete the voltage divider circuit.

p.s. Credit goes to Shilpa for teaching me how to sew.  :)

Soldering Flex Sensors

The flex sensors have really short leads, and are pretty flimsy. We decided to solder wires onto the leads to give us more flexibility in making connections. In keeping with proper EE color coding, one lead was soldered to black wire (ground) and the other to blue wire (internal connection). We did this for all 9 of our flex sensors, using the awesome soldering iron in Detkin. Automatic soldering iron cleaner FTW!

Flex Sensors

Our flex sensors came in on Wednesday, and we've spent the last couple of days trying them out.  We ordered 4 of the 4.5" sensors and 3 of the 2.2" ones, and Roop was so nice that he ordered us an extra sensor in each size.  Thanks Roop!!

Flex sensors are just variable resistors, and as you bend them, their resistance increases.  We first wanted to find out the actual resistance ranges so we hooked it up to a multimeter.  Doing it handheld (i.e. flex sensor in hand and multimeter leads hooked up) was rather difficult, so we did it using a protoboard.  We discovered that the flex sensors vary in range.  While they are close to each other, there is some variation.  For the voltage divider circuit, we chose a resistance that was in the middle of the range, so that we could get the most ADC variation.

After building a couple of test circuits, we connected the Firefly's ADC pins to the flex sensors via a breadboard.  The Firefly ADC header has 8 ADC ports (ADC0-ADC7) and two pins for power (3.3V) and ground.  Using these pins, we connected two flex sensors.  We used the same bmac_sensors program from the Firefly tutorial lab, but modified.  It now prints out the ADC values for seven sensors, named thumb, index, middle, ring, pinky, front, and side.  The first five are obvious (hopefully).  The front and side sensors are to detect if the hand is angled to the right or if it's bent forward, as some letters use these motions.

Unfortunately, the wireless transmission didn't work, so we'll have to try that later.  In the meantime though, we got the Firefly to print out the seven ADC values to the terminal, and we used that to figure out the ranges of the flex sensors.  Since each sensor had a different resistor range, we tested each one individually and numbered them.

Now that we have all that data, the next step is to make the glove, which we intend to do this weekend.

P.S. Hard data and pictures to come!

What Your SignMaster 9000 Can Do For You

SignMaster 9000

Work out your fingers

Get 9000 fingers

Tired of signing?

Get DINING

Get such a powerful hand

You'll squeeze things into SAND

Trouble signing a letter?

You'll get sooo much BETTER

Want a twist on sign language?

Try NINE LANGUAGE!

Because you'll grow 4 new fingers on each hand

Have NINE FINGERS!!!!!

On NINE HANDS!!!!!

So you can sign NINE TIMES AS FAST!!!!!!!!!!!!!!!

What's all this about?

The SignMaster 9000 is an American Sign Language alphabet decoder. This device will be embedded in a glove that can be easily worn on the signing hand. The glove will use flex sensors to analyze the unique finger configurations during the signing of a letter and display the corresponding letter on an LCD screen.

ASL is a method of communication primarily used by the deaf, with an ASL alphabet mapping each English letter to a sign language letter. The language itself is not like written English in that a word is not made by signing each individual letter that makes up the word. Instead, there is a dictionary of words that can be signed with their own unique gestures unrelated to the alphabet. Here is an image of the ASL alphabet:

For our ESE350 final project, we would like to implement a stepping stone to a complete sign language decoder. Finger spelling -- that is, signing words by spelling out the letters in the word -- is an elementary version of sign language that is primarily used in deaf education. We wish to design and build an ASL letter decoder for the use of finger spelling. Stay tuned for our technological adventures!