Archive for the ‘Mindstorm’ Tag

Lego Mindstorm Flex AIR2

This is something I dreamt up and actually finished in July but haven’t yet had time to write up a post on it. The original idea was to use Flex/AIR on the desktop to control a NXT Lego Mindstorm robot but then things got a little more ambitious by adding a bit of Augmented Reality together with real time Speech Generation.

The first step was to replace the firmware on the Mindstorm brick with LeJOS, which includes a Java Virtual Machine that allows the Mindstorm to be programmed in Java. This was a tense moment but actually went relatively smoothly.

I then created a Java application on my Macbook Pro that could control the robot via a Bluetooth link. I discovered that there is a nice little library called icommand that is designed to allow you to do just this.

Once I had that connection up and running the next step was to make sure that I could use this application as a bridge to control the Robot from an AIR application.

I therefore created an interface to my desktop Java application using Merapi, which is the same Java/AIR bridge that I used to create a previous project for Flex speech recognition.

Using this, I then created an AIR application that could send and receive Merapi messages across the bridge, which allowed to me to get sensor information back from the Mindstorm and to send messages to it for navigation and pincer/sensor control.

Having got all that working I couldn’t resist putting in some Augmented Reality, so I added an iphone to the Robot and used that as a wireless webcam to get the video feed into my AIR application.

I then adapted and incorporated my previous augmented reality projects to allow me to have interactive avatars that I could switch at runtime. These are integrated with a remote java server using BlazeDS to give me the artificial intelligence part that they need to provide realistic answers and speech.

Now around this time AIR 2 was released so I decided to create the whole thing using that instead. Instead of having separate AIR and Java Applications the whole lot is bundled together and deployed as a Native Application. This is very cool and I think adds a whole new dimension to what can be done with AIR. What actually happens is that the Air application contains on its source path, the executable java jar file for the Minstorm’s controller. Once the AIR2 application is launched it is able to launch the java application as a native process and then communicate with it across the bridge using Merapi by serialising objects between Java and ActionScript.

Advertisements