Emotiv Headset and Locked In Syndrome

In March, a salesman working for IBM had a stroke, which left him with complete paralysis, unable to use his muscles, and without the ability to speak. His brain however is working fine - a condition called Locked-In Syndrome. His means of communicating is by his eyes - looking up for yes, and down for no. He has to wait for someone to ask him if he'd like to speak before being able to do so. Then, using a letter chart, that someone must point at letters one by one, until a confirmation is received. This is the first letter of the sentence. The process must be repeated for every other letter in the sentence, until the full sentence is spelt out.

The salesman in question is called Shah, and Sarah (my wife) is his Occupational Therapist (OT), who saw the mind-reading headset from Emotiv that I was using in IBM and thought about the potential this could have for Shah. The headset was designed for the gaming industry, and measures facial expressions, excitement/boredom levels and can be trained to listen for particular thoughts, which can perform an action on the computer.

Sarah asked me to demonstrate it to the consultant, speech therapist, psychologist and hospital ward manager - who agreed it may have some potential, and who took it to the ethics committee. Shah was informed about the headset and what it can do - but that we weren't aware of any person with the symptom who had tried it before. Shah, being a bit of a techy, was up for pioneering it - so I met him last week for his first go.

Just like the many other people I have demonstrated the headset with, Shah was instantly able to train the system so that one action - the "push" action - would push the floating "Emotiv cube" into the distance. As we trained, the ability to push (and stop pushing) at will, improved.

Adding a second action adds a complication. Now the unit has to distinguish not just whether you think or don't think, but which thought you are having - much harder! And like everyone, it will take a bit of time to practise getting sufficiently good at this to control. However Shah is up for the challenge, and has a fantastically supportive family who are willing to help him train.

Last week, after one week of use, Shah had managed to train the headset very well - though when tired the Emotiv skill can go down as well as up. Working with the speech therapist, we have connected the output of the headset to the input of "The Grid 2" - a piece of software by Sensory Software which allows a user, normally via eye tracking, keyboard, mouse or switches, to control their environment, write emails and surf the internet. We have initially set up 3 different menu items, and thinking "push" starts highlighted these options in turn at 10 second intervals. Thinking push again, will select one of the options. Sounds easy perhaps - but if I tell you to *not* think of a rabbit - you can't help but think of a rabbit! It's hard to think about when to start and stop thinking, and then change to think about the thought you are supposed to be thinking about! I think this will take some practice to achieve, but once achieved could be widened to many more options, with very little extra practice.

Often people ask about using the headset for people with different conditions, and in different places. What with Shah being a fellow IBMer who likes gadgets, and his OT being my wife, and with the particular medical condition he had, conditions were ideal to try out the headset. It will take time for us all to get it right - but for now it is looking good. I will try to keep you updated!

Wearable computing devices

Time to do a bit of blogging again. I've recently been asked about various wearable computing devices... here's a starter list, but I'd be interested in any other links you may have.

AR Glasses: These plug into a phone and can overlay data like directions / emails etc whilst travelling.

Gesture Gloves: Very accurate hand movement recognition.

Gesture Watch: Accelerometer, temperature, pressure sensors - also can act as wireless hub for other sensors.

Sixth Sense (wearable video/projector): Gesture recognition, ability to detect objects and augment real objects with data by projecting back on to them. Very cool.

Emotiv Headset: Brain reading device built for gamers which reads facial expression, excitement/engagement, trained actions and head movement.

NeuroSky Mindset: Brain reading device which measures excitement/engagement. Chip has the ability to add other sensors.

Mobile Phone Jewellery: Article from 10 years ago about IBM going into jewellery that can be used as a mobile phone.


We have used some of these sensors in the ETS Lab... more in a follow up post!

Using Brain Computer Interfaces for something useful

There are now a whole collection of Brain Computer Interfaces around... and this latest one from the games company Mattel uses the power of concentration to make a ping pong ball hover. The technology is supplied by NeuroSky, who have also provided the technology for a Star Wars game - $50-$100 for the fun of using "brain waves to allow players to manipulate a sphere within a clear 10-inch-tall training tower..." - something I could imagine myself doing just the once. Never-the-less it's quite good to see the technology coming down in price.

What fascinates me however is how we might be able to use brain waves for real personal benefit and integrate it more with our every day computer interaction experience. People who have severe motor difficulties could use it as a rehabilitative aid. Performing product research, it can be used for detecting when people are excited or not when they experience a particular product or website. When we are browsing on-line, we could combine it with eye tracking to provide feedback on what really interests us, and then tailor our online experience accordingly. When we go to the shops... the cinema... driving in the car... feedback can be provided to enterprise systems in all sorts of situations, which in turn can affect the environment around us. I'm really looking forward to talking with a number of customers about how they can integrate this and other interface technology in their own innovative projects in 2009!

Calculating energy usage is harder than it seems

In my last post, I said a Christmas project was to monitor my usage - but I confess I've got a problem of how to measure my energy usage which is so far unsolved.

As I said, I have a currentcost meter (m1) which reads the current going into and out of the house. Of course, it cannot determine which way current flows, so it reads 50W regardless of whether the 50W is imported to, or exported from, my house. Therefore, I put a second meter (m2) round the wire from the solar panel, which told me how much the panel was generating and was thinking that if the value on m2 > m1 I must be exporting, otherwise I would be importing. However this thinking is flawed:

If I am generating 100W, (m2=100) and I am using 50W, then I will export 50W (so m1=50).
If I am generating 100W, (m2=100) but I am using 150W, then I will import 50W (so m1=50).

So from my two readings, I cannot conclusively say whether I am importing or exporting electricity, unless I am missing something obvious? The bright green import/export meter which Southern Electric fitted manages to work out what is being imported and what is being exported - but there doesn't seem to be a nice little serial port anywhere obvious! The other option is to buy a wireless unit as an addition to the inverter but this costs £120 and doesn't appear to give me open access to the data.

BTW - in 20 days we've generated 16.2 kWh and exported 9 of them. A grey day typically generates 200Wh the whole day - a sunny winter's day like today seems to generate about 2kWh.

The solar works

In the loft
The solar panels are connected to an inverter (right - click photos to enlarge) , which converts the DC generated current into AC, using the "only clever bit" of the entire installation, according to the installer. This is mounted just inside the entrance to the loft. The interesting thing about the blue box is the "knock interface"... to switch on the unit and light up the screen you have to knock on the grey area. The screen tells you how many watts the panels are generating, and how much have been produced during the day, and how much in total.

In the kitchen
Downstairs, next to the consumer unit is a fantastic red knob, which isolates the panels. Above it is a meter which also shows how much electricity has been generated by the panels in total. During the first day of operation, the unit generated a total of 700W - disappointing considering it was bright sunshine outside all day, and the unit is spec'd at 1000W per hour. The installation engineer reassured me that as the sun got higher during the summer months the unit would generate much more - we shall see! During the second day of operation, when it was cloudy, the unit generated a total of just 200W during the day. Not good.

Metering
The electricity meter has been switched to an import / export meter. The meter gives the amount exported to the National Grid (ie total produced by the solar panel minus total consumed at the time in the house), and also the amount imported from the Grid. You can see the current cost clamp in the picture. This is interesting, because the current cost meter measures the flow of electricity regardless of which way it flows... so it reads 100W when the house consumes 100W, and it reads 100W when the house exports 100W. It was highly confusing turning on a light to see the amount on the meter decrease! So one of my Christmas projects is to connect another meter to the solar panel cable, and add the two readings together in order to get a sensible output... more on that later.

Solar Panel installation

Here's a photo of the solar panels which were fitted to our roof today by Rayotec. They should provide us with 1kW electricity per hour in peak sunshine, but just around 100W per hour in horrible foggy weather like today. Tomorrow morning they will be grid connected, so any extra electricity we don't use will go back into the grid. With Southern Electricity giving us slightly more per unit than we pay for our electricity, there's still an incentive to use as little as we can. We expect the unit to just about cover our usage, over a yearly period. It cost around the same as having all our doors and windows replaced a few years ago and we're promised a similar length of service. So, we're just looking forward to the grand switching-on ceremony tomorrow morning!

Mind reading headsets

So far, the mind reading headset market aimed at the regular consumer is quite small... I've come across the well advertised Emotiv headset, to be released early next year, and OCZ's "Neural Impulse Activator", shown right.

Unlike the Emotiv headset, with 16 different sensors placed on different parts of the head, this unit only has 3, all placed across the forehead, inevitably giving slightly less function.

In the first several hours of playing with the device, I managed to work out what successful calibration "looked like": - when the muscle signal graph didn't vary away from the "desired baseline". Sometimes this was very easy to achieve, and other times, just didn't happen. I discovered it worked better on one side of the lounge at home, than on the other. I also concluded that wetting the 3 sensors with water helped to a degree. Unfortunately the device needs to be calibrated every time it is used, which makes demonstrations rather hit-and-miss.

My first successful experience came shortly after calibrating the device. I was able to play the game of pong, using my eyebrows. By wiggling them fractionally upwards, the bat went up, and wiggling them downwards, the bat went down. And whilst I didn't ever manage to win a whole game, I did hit the ball enough times to win a few points, so it was not all chance.

My other successful experience was linking the muscle movements to keypresses, so I could use my eyebrows to navigate around Google maps. An eyebrow up-wiggle mapped to Page Up, and sent the map flying in one direction, a down-wiggle mapped to Page Down, and sent it flying in the other. Unfortunately zooming in and out was a bit beyond me, as the '+' key was unmappable.

So how is this mind-reading? Well, unfortunately it isn't yet. Unlike the Emotiv device, it seems impossible to "train" the device to recognise when you are thinking of stroking a cat for example. As found in this review, it should be able to detect general mood - OCZ's VP Dr. Schuette, says he can think of an expletive to make his Alpha 2 increase enough to trigger a "jump" action in his computer game. However I suspect this takes many hours of practice!

So for now, I shall continue playing pong with my keyboard, and wait for a few weeks when I hope to be able to get my hands on the Emotiv headset.