Sensorium Project - 3D-360-degree Multi-touch(less) HID sensor

We've successfully interfaced our solid-state 360 degree sensor with Arduino, Teensy and Processing to control 3D Processing and Arduino java sketches & 3D objects as well as emulation of mouse, keyboard, joystick e.g. for massive online multiplay online gaming (MMOG).

Yes! we are developing interfaces to use the sensor as a 360 degree / 3D HID type of mouse !  Proof of concept is now completed (see videos). 

We are also taking this much further and developing gesture sensing applications - but there is still a long way to go to. 

Touchless sensors are seen by many companies as the future such as  leapmotion used by HP

We therefore have opened up discussions on processing  forums for some further ideas and programming support. 

Some possible development directions could also be...:
  • 3D gesture sensing, for example using the TUIO interface
  • 3D midi interface (see 3D drum machine example)
  • 3D interface to kinetic art projects 
  • 3D Touchless input to mobile device
  • Control of multi axis CNC machines and Robots


moving a hand over the sensor controls PC applications

Background to Sensorium Project

We used Arduino UNO & Teensy to control the serial communications between 360 degree HID sensor and the Arduino Uno , Teensyduino and Java / Processing sketches on the PC/Mac. The sensor controls 3D graphics (Java sketches), or can emulate Mouse, Joystick or Keyboard on the PC (or possible would also work on Apple Mac, Andriod, Ubuntu, etc but have not test this yet.) to control any 3D applications such as Google Sketchup, Google maps, AutoDesk Inventor - CAD type application.




Develop your own Arduino Sketch or Processing.org sketches
We are using Processing.org , Arduino, Teensy as a development platform as it's real easy to develop cool java graphics programs and Processing GUI  is free and really a great way to learn about programming and interfacing.

3D Mouse / Keyboard / Joystick  Control

Plug the IRCF360 sensor into a Teensy2  to enable 3D mouse, joystick, keyboard interface to your Arduino projects. The Teensy2 can be configured as a USB device such as mouse, Joystick, keyboard which allows the sensor to control your PC or Mac ....adding new dimensions to your projects.

So how does this work?...By moving your hand over the sensor the direction and distance of your hand / fingers are sensed and translated into values within the Teensy2..... 

Gaming control

Keyboard emulation interface

Sensing directions are translated into keyboard keys. This is programmed into the Teensyduinon - sketch  using the usb keyboard commands: 

Keyboard.set_modifier(0);
Keyboard.set_key1(KEY_W); 
Keyboard.send_now();

e.g. configurate of the keys WASD to control online or Massive Multiple online (MMO) games - spaceship  speed control is achieved by mapping mouse or keyboard keys to the distance readings from sensor.

Mouse emulation interface 

Sensing directions are translated to mouse directions. This is programmed into the Teensyduinon - sketch using the Mouse.move (x, y) USB mouse command ;  e.g. configurate of the keys WASD to control online or Massive Multiple online (MMO) games - engine / speed control is achieved by mapping mouse or keyboard keys to the distance readings from sensor.

Joystick  emulation interface

Sensing directions are translated to Joystick directions or commands. This is programmed into the Teensyduinon - sketch using USB Joystick command  to add Z directional control to your project too. 

The  joystick usb interface  has 
  • 32: Buttons
  • 6: Axis (X, Y, Z, Rotate Z, Left Slider, Right Slider)
  • 1: Hat Switch
Joystick.X(value); 

// "value" is from 0 to 1023 Joystick.Y(value); 
// 512 is resting position Joystick.Z(value); 
Joystick.Zrotate(value); 
Joystick.sliderLeft(value);
Joystick.sliderRight(value);

Adding 3D Control to your Java and Processing Sketches

We've opened up an open source project on various forums for support to develop your own Arduino, Teensyuino or processing sketches. 

It's easy to develop cool java scripting based graphics programs using the FREE development environment from processing.org and Arduino. Processing is free and opensource and really a great way to learn about programming and interfacing.



Using  Arduino or MAX232 serial interface to a PC the sensor can control your java and processing sketches. Examples are:   

Start writing your own Arduino or Processing.org sketeches to interface with the IRCF360 to visualise the results. The example below is the multi-blob tracking demo, where the two large blobs are tracking the closest and 2nd closest object. The brown 'spider-web' is a representation of all other directions. For more advanced Java programs & sketches the pattern could be used to interpret different hand gestures to add more advanced controls.    



Background of 3D-360 degree sensor

3D - 360 degree HID Sensor
IRCF360 'contact-free' control of a processing sketch


The IRCF360 sensor responds to commands received via the RS232 serial port (I2C will be available soon); controlling various parts of the sensor hardware and controlling the 360 degree sensing functionality as well as LED fading and ambient light sensing. 

The main commands for interfacing 360 sensing with an Arduino/Processing sketch or perhaps an Android or other App is command "30" which triggers 360 proximity sensing. 

Interface to Arduino and Processing

In the Arduino  demos we are using the Arduino's 'soft' UART for connection of the IRCF360 sensor and the USB port as the main interface to a PC and for control of of processing java sketches,  demonstrating 360 degree Sensing.  

The IRCF360 hardware will be available as a complete kit with all parts and pre-flashed firmware. If you're interested in a pre-release version, please register your interested by sending an email to support@robotmaker.eu - or by filling in the form -> here


Getting Started

Without further ado, if it tickles your fancy, please head to these webpages and processing forum where you can find all latest processing sketches or here.

We believe that with a bit more clever thinking this device can be used for more advanced things such as hand gesture detection. e.g. such as blobscanner and gesture recognition solutions 

We've released all the Processing and Ardunio sketches on the public domain. 


In the processing sketch section we've attached a few processing demo's that demonstrates the general concept and some possible applications such as tracking simple hand/finger gestures. We found that java programming enables accelerated development of graphical representations of gesture enabled applications with any platform or environment that supports RS232.


Register you interest here



If you would like us to keep you informed on the product launch or would like to purchase a pre-production version of the IRCF360 sensor just register your interest by filling in the form -> here



Next Steps

Possible areas to explore:
  • Multi-touch(less) gesture recognition e.g. using the TUIO interface .
  • swiping images based on acceleration across the sensor
  • ++



References:

TUIO enabled software Check out this extensive list of TUIO enabled software: http://www.tuio.org/?software
Gesture Works http://gestureworks.com/features/ 
Android pointing device controlhttp://developer.android.com/reference/android/view/MotionEvent.html#ACTION_MOVE 
  
 Fluid Examples http://www.openprocessing.org/visuals/?visualID=27653 
http://www.openprocessing.org/visuals/?visualID=3768
http://www.openprocessing.org/visuals/?visualID=13533
http://www.openprocessing.org/visuals/?visualID=22459


 blob detection and blobscannerhttps://forum.processing.org/topic/new-blob-detection-and-analysis-image-labelling-and-segmentation-library-for-processing  
http://www.openprocessing.org/visuals/?visualID=22459
http://www.instructables.com/id/3D-Motion-Capture/
 webcam driver for testing blobscanner WinVDIG 1.0.1 (NOT the latest version) This will hook up Processing to your webcams. 
Processing sketches  video - Radial Pocky
 

ċ
Arduino_IRCF360_030_RADAR5c_2_x_Blobs.pde
(21k)
ROBOTmaker Support,
Mar 14, 2012, 9:44 AM
ċ
Arduino_IRCF360_3D_Cube_1a.pde
(20k)
ROBOTmaker Support,
Mar 20, 2012, 4:36 AM
ċ
Arduino_IRCF360_3D_Radar_Cube_2a.pde
(19k)
ROBOTmaker Support,
Mar 20, 2012, 4:23 AM
Comments