After some period of inactivity, I’m now (finally) conducting my PhD research on Sketching Recognition here at Louvain Interaction Lab (Belgium), and my goal is to make Eclipse Sketch a repository of my contributions to that field.

At this point, I’m requesting sketches from you out there. If you are engaged on interactive systems development, please consider sending me your sketches by going to:

http://callforsketches.ugosan.org/

Anything you draw during meetings is useful. You probably have sketched diagrams into post-its, pieces of paper even napkins. Scan or photograph and send them over, itwill help a lot.

post-it with the map

There is an article called Trainable Sketch Recognizer for Graphical User Interface Design from A. Coyette and others showing an approach to recognize pen-made sketches based on Levenshtein distance algorithm for string comparison.

The article talks about recognizing elements of user interface such as buttons, combo boxes and windows, when they are sketched by a designer. Well, they do that by using Levenshtein’s String Distance algorithm. Thats right, an algorithm created for string comparison to check if they are closer or not, in terms of character swapping. You know, when there is a typo on you search at google like “algoritm”, it says “you meant algorithm”. Thats a string distance algorithm working. It is really simple and I was quite amazed about how good it worked for Sketch.

So, what the authors of the article did to transform drawings into words? First you need to assign a number for each cardinal point, to compose your words. Lets say 1 for north, 2 for northeast, 3 for east, and so on. Then take the points (x,y) of the sketch and then each pair is compared to the next one, if the point is at north of the previous one then its a gesture going up and the character relative to North its added, 1.

How do you know what cardinal point each pair of x,y belongs relative to the previous is easy, take a look at the post-it at the beginning of this post (its actually a post-it, its on my wall). Lets get two points A and B, if (B.x – A.x) its positive and (B.y – A.y) its zero, then B its at East of A. If they are negative and positive respectively, then B its at SouthWest from A.

So a square would be something like: 3333333355555557777777111111111
A triangle would be like: 45454544537777777782232312

But people draw things differently, you may start a square by moving your pen South, instead of East, for example.
Since the algorithm its so fast, you may compare your sketch to several samples of squares, several words, or even better, let the user tells your application what he/she meant with that they just drawed.
“this is a square, learn the way I do”

Its a powerful tool which combined with other algorithms such as corner finding could give a fingerprint of the user’s sketch.

This is implemented as the single one algorithm responsible for recognition on the Sketch Shapes Application. There is more about this on the way, as the project matures: new algorithms will take place, but I think it couldnt be any simpler than Levenshtein’s.

Kudos for Coyette and the team.

What do you think of this logo? (Click on it.)

Good news, the Sketch project was finally created! You may find it at http://www.eclipse.org/sketch

Although some improvements and refactoring are taking place right now, the API is already usable if you would like to pre-alpha-test it.

Me, Chris and Mariot have decided to go for Git as the version control. We are a fresh new project, so there were no reason not to: http://git.eclipse.org/c/sketch/org.eclipse.sketch.git/

Be aware that we are still re-organizing everything, and the code will be probably changed a lot, but if you see something really ugly, please file a bug.

The future of Sketch, as I envision it, includes users drawing their own graphical representations for models, being able to make suggestions to existing models using their own vocabulary and ultimately writing their own models. There is a very interesting discussion on this regard at the forum. If you are interested, please join it.

As for the present, I’ve prepared two videos for FlexiTools’2010, showing the features already available in Sketch.

1. VIDEO Creating a Use Case Diagram

Here I extend the Shapes editor (which is only able to model Squares and Triangles) to build such diagram.

2. VIDEO Creating an Activity Diagram and Making Annotations

Here is a slightly more complex example, plus annotations, which is the most immediate outcome of Sketch, since no recognition is needed.

There’s a long time since i’ve blogged about the FlexiTools’2010 workshop and the discussion about a flexible graphic representation made possible in Eclipse graphical editors.

Here we talk about creating an intermediate element which holds a flexible graphic representation, in opposition to it’s canonical one (which in GMF is defined at .gmfgraph model). The users would be able to draw the elements as they wish, and as well be able to create new ones, something like this:

Here is a video explaining how it works, please watch and comment (go to Sketch Forum):

I’ve just submitted a position paper to FlexiTools’2010, in which me and Simone outline our ideas for the Sketch API, which i’m carrying on with Chris and Mariot.
Take a look at the paper:SKETCH: Modeling Using Freehand Drawing in Eclipse Graphical Editors

The main idea is the approach of the modeling task as a twofold process — one of freely sketching models with little interruption from the system; and another more formal, “classic modeling”, recognizing the elements drawn by the user right away, having as output a model in its canonical representation.

By letting users draw the model using their own graphic representation, we are allowing the GEF/GMF editor to be more flexible regarding its visual notation. For instance, assuming a simple model with the elements named flower, sun and cloud – they might be connected and generate an output or might serve as input for other models. A user might draw the flower element, for instance, in infinite ways, having any number of petals, with or without stem, and so on.

This allows the creation of a “graphicless model”, without predefined visual counterparts to the model elements, just elements and relations — the user would choose how elements will look like. That means the .gmfgraph would just hold the canonical representation, but the real one will be the user’s.

 So, on Eclipse GEF/GMF frameworks side, some minor modifications will need to take place:

• The underlying model (Ecore?) will need a generic element to serve as an ‘unrecognized’ element, to be created at the model while the user does not signifies it as anything
• make the editor flexible enough to hold any graphic representation for it’s elements, representing it using an SVG with the sketch. Also, this approach can also be used to make annotations on existing models, since the user might be able to create an “annotation” element.

Those are all feasible, so little adaptation would be needed to plug sketch onto GEF/GMF editors

Don’t you think?

3 weeks ago Sketch API proposal became official. Meanwhile, our job is to foster community around the project.

This project is intended to be applied to any GEF editor, and since that there are so many GEF-based graphical editors out there, our goal now is to try to define a common set of functionalities between them.

We will start to discuss some of the ideas using the Sketch Forum/Newsgroup, please join the discussion and create topics if you are interested on this project or if you know a GEF editor that this project could benefit.

So far, what we have in mind is:

• Gesture (not only shape) recognition – the API would be able to provide a set of configurable gestures, that would translate into commands for moving, resizing, and so on. This feature would massively benefit GEF3D.
• Sketched vs. ‘Beautified’ look – There would be two ways of presenting the diagrams: the Sketched would show the recognized element as the user sketched it (using an SVG Figure created from it); The ‘beautified’ look would be the EditPart’s normal representation.
• Postponed recognition – Maybe to have an Action to trigger the recognition would be better for some domains, instead of always asking the user what he meant to represent like on this demo.

Imagine to have a mockup/wireframe editor built on top of Visual Editor. A prototype builder might be coded using the user’s own drawings. And the prototype might be constructed without a ‘done look’.

Any more ideas?

We are also editing the Sketch proposal’s Wiki so the modifications would be merged onto the official one later. If you are an interested party, please add yourself there.

Pen-based/Tablet technology it’s been around for a long time, but recently this ‘touch-oriented interaction’ has been gaining more attention, with all the buzz around iPad. So if we’re starting to interact to devices with our hands, we are not clicking anymore — perhaps our interfaces should start to be less click-oriented? How can we take advantage of the popularity of those devices? Wouldn’t be nice if we had sketching recognition withing Eclipse?

I think so, I’ve been working on sketch recognition for GEF/GMF editors while creating the MoLIC editor, take a look:

Sketch API (former OmniModeling) on MoLIC Editor from Ugo Sangiorgi on Vimeo.

There are other posts on this blog reporting the evolution until this point — the API is able to recognize shapes the way you draw, learning withing the process if it cant recognize it upfront. It can recognize any shape, really, you will ‘teach’ it how.

Omnimodeling: Multi-Agent Recognition from Ugo Sangiorgi on Vimeo.

So it’s on! I decided to contribute to Eclipse by proposing a Sketch API for GEF/GMF using the former OmniModeling project base, take a look at the proposal we are working on. Mariot Chauvin and Chris Aniszczyk are helping me through, thank you so much guys!

Comments and contributions are welcome

I should acknowledge Simone D.J. Barbosa for the kind support during the OmniModeling project, as well as my colleagues at the Semiotic Engineering Group of PUCRio.

What if I give to GEF editors the ability to recognize sketches, using the Levenshtein algorithm, which is easy to implement and its also fast?

Instead of clicking on the palette, the user just select the Free Drawing Tool and sketch freely on the editor. A given layer could recognize the points and add the relative element to the diagram.
Also, if the recognizer didnt get the sketch upfront, there could be a mechanism that let the user tell the program what he/she meant with that sketch, lets say a popup menu that let the user choose what element is meant by the sketch.

This could be a nice feature to have on GMF. You model your domain, works the Graphical model, the Tooling model, and there could be also a Sketch model for you to create, binding gestures to elements.

I’m starting a prototype. I tought it might be called OmniModeling. The modeller of everything