Somebody asked in StackOverflow how to do smooth movements of objects (widgets) in python game platforms. I didn’t have the time to reply there because the question was closed because the question was posted in a way that promotes debate not answers. But here is an answer. I think. I hope. I just wanted to say that in Kivy is possible to do smooth movements, not complicated, very flexible and there is a tutorial about it: Pong Game Tutorial

This is a simplified code based of the tutorial where I just make the bool move slowly in diagonal direction from bottom left to top right. I hope it will help somebody.

from kivy.app import App
from kivy.uix.widget import Widget
from kivy.properties import NumericProperty, ReferenceListProperty,\
    ObjectProperty
from kivy.vector import Vector
from kivy.clock import Clock
from kivy.lang import Builder

Builder.load_string("""
<PongBall>:
    size: 50, 50 
    canvas:
        Ellipse:
            pos: self.pos
            size: self.size

<PongGame>:
    ball: pong_ball

    PongBall:
        id: pong_ball
        pos: 0,0
""")

class PongBall(Widget):
    velocity_x = NumericProperty(1)
    velocity_y = NumericProperty(1)
    velocity = ReferenceListProperty(velocity_x, velocity_y)

    def move(self):
        self.pos = Vector(*self.velocity) + self.pos

class PongGame(Widget):
    ball = ObjectProperty(None)

    def update(self, dt):
        self.ball.move()

class PongApp(App):
    def build(self):
        game = PongGame()
        Clock.schedule_interval(game.update, 1.0 / 60.0)
        return game

if __name__ == '__main__':
    PongApp().run()

So yesterday I posted that I was puzzled by the way Kivy deal with transformations (Rotate, Translate and Scale). Especially, the  way RelativeLayout relates with all of this. Please read my previous post first: Rotate, Translate, Scale and RelativeLayout in Kivy .

I actually asked in StackOverflow and I get an insight and I think I am about to understand what is happening here.

This involves 2 Kivy basic instructions (PushMatrix and PopMatrix) that I completely ignored before. Their names are unfortunately very misleading from the way I am seeing things. Here is a what the kivy documentation indicates:

  • PushMatrix: “PushMatrix on context’s matrix stack” [1]
  • PopMatrix: 2Pop Matrix from context’s matrix stack onto model view” [2]

I don’t think that is very helpful. This has something to see with OpenGl which I have never worked with and it is probably the source of all my misunderstandings here.

So I ll try to put things in the way I see it. I understand that Push and Pop are operations in a Stack [3]. But what are they pushing and popping? They push and pop the current context of the canvas. Let’s see the example first. In my previous post I had two options to solve my transformation: undo all the transformations and use RelativeLayout. Here is a 3rd option using push and pop.

Option 3 – Using Push and Pop:

<Example>:
    Button:
        pos: [0,0]
        size_hint: [None, None]
        size: [200,200]
        canvas.after:
            PushMatrix:
            Line:
                circle: [50, 100, 50]
            Translate:
                x: 100
            Line:
                circle: [50, 100, 50]
            PopMatrix:
    Button:
        pos: [200,0]
        size_hint: [None, None]
        size: [200,200]
        background_color: [0,0,1,1]

So the PushMatrix and PopMatrix actually push and pop the original context. Moreover, this is what RelativeLayout is doing in its internal code, together with an expected translation to the position of the widget.

So what is this matrix? I resisted to believe that the matrix is a drawing space of the size of the screen. For example, a matrix of 800×600 that we are popping and pushing constantly. The Kivy documentation explains it here and I think I got the answer. It reminds me my course on linear Maths 10 years ago.

The idea is that you can save many transformations in a simple matrix and then applied to any vertex instruction (lines, polygons, etc) you were about to draw. For example, before you are going to display a Line,you apply (by mathematical matrix operators) the matrix to the points of the line. A matrix can save many transformations, I honestly don’t remember if all of them but surely can rotate, translate and scale figures.

I started to see my general problem here. A Kivy canvas is not a canvas like when you paint but this will be another post.

[1]  http://kivy.org/docs/api-kivy.graphics.html?highlight=pushmatrix#kivy.graphics.PushMatrix

[2] http://kivy.org/docs/api-kivy.graphics.html?highlight=popmatrix#kivy.graphics.PopMatrix

[3] http://en.wikipedia.org/wiki/Stack_(abstract_data_type)

I have been puzzled by the way Kivy deal with transformations (Rotate, Translate and Scale). The Kivy documentation says that “each widget in Kivy already have by default their Canvas"[1] but that doesn’t seem the case. At least in practice, it seems that all widgets share the same canvas as if it were a Reference.

Here is the very basic main.py

# main.py
from kivy.app import App

class Example(Widget):
    pass

class ExampleApp(App):
    def build(self):
        return Example()

ExampleApp().run()

Now we have this very simple python code. There are 2 buttons, both of 200×200 pixels. I want the first next to each other, so I position the first in the (0,0) coordinate and the second one in the (200,0) coordinate. Also, I want to circles in the first Button so I use the canvas.after for that. I draw one circle of radio 50px, translate the canvas of the Button (which is a Widget) 100px (the diameter of the first circle), and draw exactly the same circle again after the tanslation so I have one circle next to each other.

<Example>:
    Button:
        pos: [0,0]
        size_hint: [None, None]
        size: [200,200]
        canvas.after:
            Line:
                circle: [50, 100, 50]
            Translate:
                x: 100
            Line:
                circle: [50, 100, 50]
    Button:
        pos: [200,0]
        size_hint: [None, None]
        size: [200,200]
        background_color: [0,0,1,1]

I executed this with python main.py –size=500×100. The following Figure shows my result.

one

How come there is a black space in between both of them? I was expecting for this instead:

two

The Translate is affecting the whole drawing space of the window. This is a very control environment so we can guess what is happening but in real scenarios things can get completely misleading. A rotation could be the end of your interface. At this point we have two options:

Option 1 – Undo all the transformations:

Make sure to undo all the transformations we applied to the canvas. This is why it seems that the canvas is share by all the Widgets.

<Example>:
    Button:
        pos: [0,0]
        size_hint: [None, None]
        size: [200,200]
        canvas.after:
            Line:
                circle: [50, 100, 50]
            Translate:
                x: 100
            Line:
                circle: [50, 100, 50]
            Translate:
                x: -100
    Button:
        pos: [200,0]
        size_hint: [None, None]
        size: [200,200]
        background_color: [0,0,1,1]

Option 2 – Use the RelativeLayout:

If we embed the button inside the RelativeLayout, our problems are also solved.

<Example>:
    RelativeLayout:
        pos: [0,0]
        Button:
            pos: [0,0]
            size_hint: [None, None]
            size: [200,200]
            canvas.after:
                Line:
                    circle: [50, 100, 50]
                Translate:
                    x: 100
                Line:
                    circle: [50, 100, 50]
    Button:
        pos: [200,0]
        size_hint: [None, None]
        size: [200,200]
        background_color: [0,0,1,1]

This brings me to a question I am still trying to find. What exactly a RelativeLayout does? The documentation talks about relative positioning [2] but I was not expecting the impact in the whole transformation sets of Rotate, Translate and Scale. The first two could be considered positioning in certain way. Not that clear for scaling.

EDIT: I found and answer for this and posted it here. It leads to another option:

Option 3 – Using PushMatrix and PopMatrix

 

[1] http://kivy.org/docs/api-kivy.graphics.html

[2]http://kivy.org/docs/api-kivy.uix.relativelayout.html?highlight=relativelayout#kivy.uix.relativelayout.RelativeLayout

Final Post: Gamex and Faces in Baroque Paintings

Face recognition algorithms (used in digital cameras) allowed us to detect faces in paintings. This has gave us the possibility of having a collection of faces of a particular epoch (in this case, the baroque). However, the results of the algorithms are not perfect when applied in paintings instead of pictures. Gamexgives the chance to clean this collection. This is very important since these paintings are the only historical visual inheritance we have from the period. A period that started after the meet of two worlds.

1. Description

Gamex was born from the merging of different ideas we had at the very beginning of the Interactive Exhibit Design course. It basically combines motion detection, face recognition and games to produce an interactive exhibit of Baroque paintings. The user is going to interact with the game by touching, or more properly poking, faces, eyes, ears, noses, mouths and throats of the characters of the painting. We will be scoring him if there is or there is not a face already recognized on those points. Previously, the database has a repository with all the information the faces recognition algorithms have detected. With this idea, we will be able to clean mistakes that the automatic face recognition has introduced.

The Gamex Set
The Gamex Set.

2. The Architecture

A Tentative Architecture for Gamex explains the general architecture in more detail. Basically we have four physical components:

  • A screen. Built with a wood frame and elastic-stretch fabric where the images are going to be projected from the back and where the user is going to interact poking them.
  • The projector. Just to project the image from the back to the screen (rear screen projetion).
  • Microsoft Kinect. It is going to capture the deformations on the fabric and send them to the computer.
  • Computer. Captures the deformations send by the Kinect device and translates them to touch events (similar to mouse clicks). These events are used in a game to mark on different parts of the face of people from baroque paintings. All the information is stored in a database and we are going to use it to refine a previously calculated set of faces obtained through face recognition algorithms.

3. The Technology

There were several important pieces of technology that were involved in this project.

Face Recognition

Recent technologies offers us the possibility of recognizing objects in digital images. In this case, we were interested in recognizing faces. To achieve that, we used the libraries OpenCV and SimpleCV. The second one just allowed us to use OpenCV with Python, the glue of our project. There are several posts in which we explain a bit more the details of this technology and how we used.

Multi Touch Screen

One of the biggest part of our work involved working with multi-touch screens. Probably because it is still a very new technology where things haven’t set down that much we have several problems but fortunately we managed to solved them all. The idea is to have a rear screen projection using the Microsoft Kinect. Initially though for video-game system Microsoft Xbox 360, there is a lot of people creating hacks (such as Simple Kinect Touch) to take advantage of the abilities of this artifact to capture deepness. Using two infrared lights and arithmetic, this device is able to capture the distance from the Kinect to the objects in front of it. It basically returns an image, in which each pixel is the deepness of the object to the Kinect. All sorts of magic tricks could be performed, from recognizing gestures of faces to deformations in a piece of sheet. This last idea is the hearth of our project. Again, some of the posts explaining how and how do not use this technology.

The Gamex Set
The Gamex Set

Games

Last but not least, Kivy. Kivy is an open source framework for the development of applications that make use of innovative user interfaces, such as multi-touch applications. So, it fits to our purposes. As programmers, we have developed interfaces in many different types of platforms, such as Java, Microsoft Visual, Python, C++ and HTML. We discovered Kivy being very different from anything we knew before. After struggling for two or three weeks we came with our interface. The real thing about Kivy is that they use a very different approach which, apart from having their own language, the developers claim to be very efficient. At the very end, we started to liked and to be fair it has just one year out there so it will probably improve a lot. Finally, it has the advantage that it is straightforward to have a version for Android and iOS devices.

4. Learning

There has been a lot of personal learning in this project. We never used before the three main technologies used for this project. Also we included a relatively new NoSQL database system called MongoDB. So that makes four different technologies. However, Javier and me agree that one of the most difficult part was building up the frame. We tried several approaches: from using my loft bed as a frame to a monster big frame (with massive pieces of wood carried from downtown to the university in my bike) that the psyco duck would bring down with the movement of the wings.

It is also interesting how ideas changes over the time, some of them we probably forgot. Others, we tried and didn’t work as expected. Most of them changed a little bit but the spirit of our initial concept is in our project. I guess creative process is a long way between a driven idea and the hacks to get to it.

5. The Exhibition

Technology fails on the big day and the day of the presentation we couldn’t get our video but there is the ThatCamp coming soon. A new opportunity to see users in action. So the video of the final result, although not puclib yet, is attached here. It will come more soon!

[youtube BVYq_cBf8z4]

6. Future Work

This has been a long post but there is still a few more things to say. And probably much more in the future. We liked the idea so much that we are continuing working on this and we liked to mention some ideas that need to be polished and some pending work:

  • Score of the game. We want to build a better system for scores. Our main problem is that the data that we have to score is incomplete and imperfect (who has always the right answers anyway). We want to give a fair solution to this. Our idea is to work with fuzzy logic to lessen the damage in case the computer is not right.
  • Graphics. We need to improve our icons. We consider some of them very cheesy and needs to be refined. Also, we would like to adapt the size of the icon to the size of the face the computer already recognized, so the image would be adjusted almost perfectly.
  • Sounds.  A nice improvement but also a lot of work to have a good collection of midi or MP3 files if we don’t find any publicly available.
  • Mobile versions. Since Kivy offers this possibility, it would be silly not to take advantage of this. At the end, we know addictive games are the key to entertain people on buses. This will convert the application in a real crowd sourcing project. Even if this implies to build a better system for storing the information fllowing the REST principles with OAuth and API keys.
  • Cleaning the collection. Finally, after having enough data it would be the right time to collect the faces and have the first repository of “The Baroque Face”. This will give us an spectrum of how does the people of the XVI to XVIII looked like. Exciting, ¿isn’t it?
  • Visualizations. Also we will be able to do some interesting visualizations, like heat maps where the people did touch for being a mouth, or an ear, or a head.

6. Conclusions

In conclusion we can say that the experience has been awesome. Even better than that was to see the really high level of our classmates’ projects. In the honour of the truth, we must say that we have a background in Computer Science and we played somehow with a little bit more of adventage. Anyway, it was an amazing experience the presentation of all the projects. We really liked the course and we recommend to future students. Let’s see what future has prepared for Gamex!

Some of the projects
Some of the projects

This post was written and edit togetter with my classmate Javier. So you also can find the post on his blog.

Kivy is an open source library for rapid development of applications that make use of innovative user interfaces, such as multi-touch apps”. It works very well except that the documentation is not fantastic. The introductory tutorial is excellent but after that things start to get complicated.

Specifically, Javier and me struggled with the design of the interface. After a weekend reading the documentation and even code, we finally came with a solution that still feels sort of hacked, specially the coordinates (x,y) of the events. However, there were also nice surprises that start to explain some (at the beginning) weird decisions in the kivy arquitecture.

Since it is difficult to enumerate all the mysteries that we solved out, I will got through the code. The first important thing that you should know about Kivy is that it is not merely python. They have their own Kivy language. Even though, it is possible to use all the library with python, there are several advantages of the Kivy languange that will save you lots of code.

The kivy code is saved in files .kv. To use this code, all you need is

from kivy.lang import Builder
Builder.load_file ('path/to/file')

The other important thing is to distinguish between Apps, Widget, Canvas and Graphics.

  • App is generally one and will control your program. It needs at least one Widget.
  • Widgets. I would divide this in three types:
    • Form components: any button, label, textfield or similar
    • Layouts: these are basically panels in which you can sort other widgets (AnchorLayout, GridLayout, BoxLayout, etc)
    • Image: in our particular application, the most important one
  • Canvas. Here is where the confusion begins and I am still not sure if I am totally right. It seem that there is just one canvas and it corresponds to the “drawable” part of the App (the app, not the widget). And this brings problems because when you capture events it could be produce in any area of the App (except the areas occupy by widgets that captures their own events such as buttons).  The problem is that you have to calculate your own relative coordinates from the absolute ones.
  • Graphics. This are objects that you can draw in the canvas: rectangles, ellipses, etc.

A minimal application will look like this

from kivy.app import App
from kivy.lang import Builder
from kivy.uix.floatlayout import FloatLayout
Builder.load_file('gamex.kv')
class Controller(FloatLayout):
    # Class attributes....

def __init__(self, **kwargs):
    super(Controller, self).__init__(**kwargs)
    #some initial configuration but be very careful
    #because kivy is quite sensitive with the constructor
    #use the .kv instead

class ControllerApp(App):
    def build(self):
    return Controller(info='Hello world')

if __name__ in ('__android__', '__main__'):
    ControllerApp().run()

The file gamex.kv is actually the one that organize the interface which you can see in the next picture. Some details are missing but basically it has three components. The image, a right-side menu and a bottom menu.

I will try to give you some clues of the code:

  • Line 1: it’s mandatory and specify the kivy version.
  • Line 3: it has to have the same name of the first widget of the application. Look in the previous python minimal example. The rest of the code is indented inside of this line so it belongs to controller
  • Line 4 to 7: The right side of the assignations must be attributes in the class Controller. The second part of the assignation are ids for some of the components of the interfaces. I will be back to this at the end.
  • Line 9 to 11: This correspond to the image component of the interface. Anchor layout means that the position is going to be set by two of five types of references: top, bottom, right, left and center. One of the references set the x axis and the other the y axis. So in total there are nine combinations. In the example the layout is going to use the top-left part of the widget.
  • Line 15: We can reference the attributes of the class using the variable root. The nice thing is that if the attribute change, then the interface is updated automatically.
  • Line 16: Complements the previous 9 to 11. The size_hint of the image (not the AnchorLayout as you would expect) is the one that stablish the size of the Layout.
  • Line 19 to 40 are not important.
  • Line 43: This is going to define the right list of buttons. The size hint is set in Line 49
  • Line 47: Stablish a BoxLayout to organize the buttons. In this case the buttons are actually kivy images. The problem we found with the buttons is that if you assign and image to a button, then the image is distorted.
  • Line 96:  It has a very similar structure to line 43 and correspond to the bottom components. In this case we use buttons and labels since we didn’t need to put images.

This kivy languages already have saved us a lot of code in our interface but also there is another important detail that I need to explain. It corresponds to lines 4 to 7. As I said these assignations corresponds to attributes inside the class and ids inside the kivy code. So, for example, the line 4 assigns the id _paint to the attribute paint (the attribute in the final code is here). The _paint is in line 14 and also you must add the attribute to the class. The previous python code will look like this:

class Controller(FloatLayout):
# Class attributes….
paint = ObjectProperty(None)

I guess that’s it. I didn’t expect to say that much. Thanks.