Perlin noise has all kinds of applications, from realistic looking texutres to fire effects to flow fields. It’s a useful tool to have in the belt.
If you’ve never heard of perlin noise here’s the wikipedia article and here’s a great explanation of how it’s calculated. Better than that, here are some examples of its use (images are links to the original posts they are sourced from):
Natural Textures
Procedural Terrain Generation
Flow Fields
I wanted to have some perlin noise I could use to apply different effects to GPUImage filters. I found a reference to a GPU implementation on stackoverflow.com. The implementation on github here is ready to go. I used the classic 2D implementation provided there. There are also 3D, 4D as well as simplex versions (the simplex noise performs better at higher dimensions).
Here’s that code used to create a filter that generates a texture using perlin noise. The code is just copied from the github implementation:
If you read my previous tutorial on how to create a custom GPUImage filter there shouldn’t be any surprises here. I’m not going to go over how the code creates the noise, because I haven’t fully investigated how it works myself. I don’t really care how it works, I’m only really interested in how to use it.
The usage lies in the cnoise(vec2) function. You give it an x and y coordinate, and it returns a number between -1 and 1. In the filter, I’m scaling up the input coordinate (because textureCoordinate is between 0.0 and 1.0 so that won’t give us very interesting noise). Then I’m adding 1 and dividing by 2 to convert from a -1 to 1 range into a 0 to 1 range (which is the range of our color values).
Finally, I use the start and finish color values supplied to the uniforms to create a noise texture that ranges between those two values. Here’s what the result looks like (I’m using white and black as start and finish color values, but you can set it to any two colors):
You can use this filter with a blend filter to create interesting texture. You could, for example, combine it with a brightness filter, and then put a sepia filter on it. This would create a natural faded look, something like this:
However, for me, the more interesting usage of this perlin noise function is to combine it with other functions. You can use this function in any of your shader code to create interesting, natural randomness. A modified Pixellate filter that doesn’t create square pixels, but uses perlin noise to determine the size of the pixel cutoff. Here’s a video of the result:
The code is mostly the pixellate filter. I’ve added a scale uniform and of course the perlin noise function to the shader code:
#import "GPUImageFilter.h"
@interface GPUImagePixellateCrazyFilter : GPUImageFilter
{
GLint fractionalWidthOfAPixelUniform, scaleUniform;
}
// The fractional width of the image to use as a size for the pixels in the resulting image. Values below one pixel width in the source image are ignored.
@property(readwrite, nonatomic) CGFloat fractionalWidthOfAPixel;
@property(readwrite, nonatomic) CGFloat scale;
@end
The applications of perlin noise are endless. I just wanted to throw a couple things out there to spark some imagination. Here is another great creative coding applications (by @inspirit) that uses perlin noise to guide a particle system to paint an image with ‘strings’, this stuff is awesome:
Click on the above to go to the site and play with the demo.
In this post we’re going to cover the basics of how to write an image processing filter using glsl. We’ll be producing a variant of an existing filter. Here’s a look at our final project:
We’ll be using the GPUImage framework in order to jump in to writing glsl shaders and avoid all the boilerplate misery that OpenGL requires. You can get the sample project here.
GPUImage Framework is Awesome!
I’ve been on this quest now for several months to learn OpenGL. As I stated in my previous post, I got really interested in image and video processing because of my work with Core Image for Ray Wenderlich’s book (iOS 5 by Tutorials). As I was thinking about how to use the GPU to do real time video processing, Brad Larson came along and released his GPUImage framework on Github.
The advent of this framework was like something out of The Secret. GPUImage was exactly what I wanted, but was not quite skilled enough to create myself. I felt as though I had summoned it from the great universe. (I’ve been working on summoning a million dollars, but so far, no dice!)
GPUImage is an open source framework aimed at the same task that Core Image is for. The advantages of GPUImage over Core Image:
GPUImage is faster than Core Image
GPUImage allows you to write your own filters (Core Image allows this on OS X, but not on iOS yet.)
GPUImage uses GLSL instead of a proprietary language
GPUImage is open source
The only drawback is that GPUImage doesn’t yet include all the filters that Core Image offers. This will only be true for a while longer, however, as GPUImage has new filters added each week. It already offers some really cool filters that Core Image does not (like the epically awesome Kuwahara filter that makes a photo look like an oil painting).
Learning OpenGL from the Ground Up
One thing I’ve realized as I’ve attempted to learn OpenGL is that I don’t learn very well starting at the beginning and working my way up the conceptual ladder. This is the way we’re taught in school most often, but I suspect of the effectiveness of it. You can think of this approach as learning to play the piano by starting with scales and dexterity exercises. It’s boring.
GPUImage is a good way to learn about OpenGL because all the low level, boilerplate code is already written. You can jump right to the fun stuff (writing new filters) and in a short time see the results. Then, if you’re interested, you can start adjusting the lower level stuff as you see the need.
I’ve sat through lectures about how OpenGL is a state machine and how matrix math works and I wasn’t able to pay attention long enough to get to the good stuff (actually building something). I’ll confess, I still don’t have a great grasp of matrix math, but I can understand how a matrix can be used to rotate, scale, and distort a primitive, and all that stuck only when I started playing with it.
There are three resources that I would suggest referring to, or reading, if you get lost at any point here. Jeff LaMarche has a series on OpenGL that’s very well written and approachable. Ray Wenderlich has a number of tutorials on OpenGL ES 2.0 (anything Ray does is very easy to follow along with). Finally, there’s a more technical resource on OpenGL here that has some detail that the previous two do not have. If mine is your first article, I would suggest reading Ray’s stuff first (at a minimum), because you will need some context to understand some of what I refer to here.
In this post I’m going to go putting together a simple app that captures live video, runs it through a filter, and displays it on the screen. Also, I’ll be going over the basics principles needed to write your own GLSL based filter using the polar pixellate filter as an example.
There are lots of other things you can do with the framework including recording video, playing video files and filtering them, and loading and saving still images. The project has good sample code showing you how to do all these things in the examples section. Thanks go to Brad Larson for creating an awesome framework and for providing some great code samples. It’s really easy to break into it.
Basic Concepts of OpenGL ES 2.0
OpenGL is a standard for drawing three dimensional worlds onto a computer screen (two dimensional). In order to accomplish one supplies three dimensional geometry (the model) color, textures, lighting, and other information to the hardware which translates the scene into something represented on screen. OpenGL added the programmable pipeline which replaced certain parts of the process with code that can be modified by the user. This gave rise to many interesting lighting, animation, and other visual effects that were difficult or impossible before. One application of the programmable pipeline is the ability to apply filters to photos or video. These programs are called shaders.
Shaders are small programs written is GLSL (OpenGL Shading Language), which looks very much like c, that are run in parallel for each unit (vertices or fragments) of the data. The first is a vertex shader and here we can alter the standard behavior of the vertices to create different effects. One commonly seen effect is to create waving grass. The shader is passed in vertices (the units that make up polygons) and can be programmed to animate the positions of the vertices according to some desired behavior.
GPUImage Vertex Shaders
Most of the time with image manipulation we are working with two dimensional unit. We won’t be doing very much with the vertex shader portion of the pipeline. The image is displayed on a ‘quad.’ A quad is just a rectangle that is drawn as a surface on to which we project the image. This is necessary in OpenGL, everything exists in three dimensional space. If we want to draw something, we first have to create a surface to draw it on to. OpenGL ES 2.0 can only draw triangles (well, it can also draw points and lines, but we can’t draw rectangles), so a quad is constructed by drawing two triangles.
The standard vertex shader in GPUImage looks like this:
There are three kinds of variables that get passed into a shader: attributes, varyings, and uniforms. Attributes are passed in per vertex. Attributes usually contain information like the vertices of the geometry we’ll draw, the vertices of the texture that we’re mapping to those geometries, color of the vertices, etc.
Varyings are variables that are declared and set in the vertex shader that get passed into the fragment shader. When they exist at the vertex level, there is a value for each vertex. When passed to the fragment shader, there are potentially many fragment values for each vertex value, the varying variable will get interpolated so that each fragment has a different value based on it’s proximity to the relevant vertices.
Uniforms are passed in by us exist at the program level. They are same across all vertices and fragments.
In the GPUImage framework we pass in two quads. The first quad is the geometry or the surface we’ll be creating, the second is the set of coordinates that we’ll use from the texture.
Basically, in OpenGL we are drawing geometric forms in three dimensional space. Then we are mapping portions of a texture (a supplied image) to those geometric forms. I’m not going to explain all the details of texturing because it has been covered very well previously. If you need to know more, I recommend the resources at the beginning of this post.
The vertex shader is usually very simple. We have a rectangular texture and a rectangular geometry, we’re just mapping the texture to the geometry and passing it through to the fragment shader. There are a couple of exceptions to this where we will do some other things in the vertex shader, but in those cases we are just trying to precalculate a few things using the vertex shader for performance reasons. In those cases we are still just passing the image through to the fragment shader to do all the real magic.
Because most of the time we’ll use this vertex shader, we aren’t going to include it in our filter subclass. It’s the default vertex shader for the GPUImageFilter class. I just wanted to show it to you so you know that there is a vertex shader always running, but we don’t need to concern ourselves with it.
You’ll notice that we create an ‘textureCoordinate’ vec2 type. This is passed to our fragment shader. In the fragment shader it will be available to us for every fragment (you can think of a fragment like you do pixels, they don’t always map 1:1 to the physical pixels on our phone’s screen, but they act like pixels). So, we’ll be using that variable value a lot. A vec2 is a pair of points like a CGPoint. We’ll also be dealing with vec4 quite a bit, for example, when we manipulate color (r, g, b, alpha).
The gl_position = position maps our input attribute of the geometry position to the internal variable gl_position. We must write to gl_position in the vertex shader for the geometry to get into the internal pipeline. There is another required internal variable in the fragment shader called gl_FragColor. This variable is what we right to in order to control the color of each fragment.
All of the above is just background really. I could go over how the GPUImage framework sets up the framebuffer objects and how it sets and controls state in OpenGL, but the truth is, you don’t need to worry about it in order to write your own filter. This is what’s great about it.
We’re going to start writing our own filter class. When we’re done we’ll create a sample view controller class to filter the incoming video and filter the result.
Setting up a new GPUImage Project
We’ll need to set up a new project. Go to File->New->Project->Single View Application. Name the project whatever you want. You can tick (or leave ticked) storyboards and ARC. Make the project iphone only.
Right click on the frameworks folder and choose Add Files to “whateverprojectname.” Navigate to and choose the GPUImage project. Now we need to add the target as a dependency and add a bunch of frameworks and the GPUImage library to the Link Binaries with Library section. Add the following frameworks so your screen looks about like this:
Finally, in the build settings, we need to add -ObjC to the other linker flags and put the location of the GPUImage/framework folder in the header search paths. You should now be all set up.
Now we’re ready to write our own class. We’re going to re-create the polar pixellate shader and add a twist. We’ll add posterize functionality to it as well. Posterize reduces the number of colors displayed.
Polar Pixellate Posterize
The filter uses polar coordinate system to pixellate an incoming image.
The first thing to do is create a new subclass of GPUImageFilter. Call the subclass GPUImagePolarPixellatePosterizeFilter.
Add the following code to the header file:
#import "GPUImageFilter.h"
@interface GPUImagePolarPixellateFilterPosertize : GPUImageFilter {
GLint centerUniform, pixelSizeUniform;
}
// The center about which to apply the distortion, with a default of (0.5, 0.5)
@property(readwrite, nonatomic) CGPoint center;
// The amount of distortion to apply, from (-2.0, -2.0) to (2.0, 2.0), with a default of (0.05, 0.05)
@property(readwrite, nonatomic) CGSize pixelSize;
@end
We’re going to pass two uniform variables into this filter. The center variable is the point from which the pixellate polar calculation originates. You’ll see that the default is 0.5, 0.5. This is the center of the screen. The coordinate system within OpenGL ranges from 0.0, 0.0 to 1.0, 1.0 with the origin in the bottom left hand corner. The pixellate value controls how much (how big) the pixel cones are. Because we’re using a polar coordinate system, the x value is the radius (distance from ‘center’) and the other is an angle in radians.
Let’s take a step back . . . in order to set up a new filter, we need a couple of things. By subclassing the GPUImageFilter class we’ll get all the apparatus that sets up the OpenGL framebuffers and state. But, we’ll need to actually write the shader itself, and create the pointers for any uniforms that we want to include. We’ll also create class variable to store the values that we pass into the uniforms. We need both the GLint that’s a pointer to the uniform and an instance variable to contain the data for that uniform (this is for convenience). We could set the uniforms by hand, but why would we do that?
Our First Shader
On to the first part of the implementation file. Add the following code before the @implementation line:
The shader code itself is enclosed in the SHADER_STRING() macro. We assign the shader string to a const NSString object. This const will then be given within the -init call to set up the filter object. There’s a whole lot of compiling and error checking and sending the shader to the GPU that we don’t have to worry about here. If you want to know what all is happening, you can read the Jeff LaMarche post about it here. GPUImage (and Cocos2d 2.0) use the GLProgram class that Jeff Lamarche sets up in that post (both with some modifications).
Let me say a few words about data types and operations in GLSL. The main data types are int, float, vector (vec2, vec3, vec4) and matrix (mat2, mat3, mat4). You can perform matrix and vector operations using simple arithmetic, ie. vec2 + vec2. Also, floats or int combined with complex types will also work (float * vec2 = vec2.x * float, vec2.y * float). A complete list of types can be found here. (about half way down) Finally, you can do things like ‘vec4.xyz’ to get a vec3.
Let’s go through this shader line by line. The first four lines set up the variables. textureCoordinate is the vec2 that is set in the default vertex shader we looked at earlier. The center and pixelSize uniforms are the variables we’ll set up in the class itself. I’ll show you how to set those values in a minute. Finally, we have the inputImageTexture variable that’s type sampler2D. This uniform is set by the superclass, GPUImageFilter, and will be whatever image we supply to the filter class. It could be a still image or a frame from incoming video (either from the device’s camera or a movie file). Sampler2D is a two dimensional texture data type.
You may have noticed that we are using the highp qualifier all the time. We have to tell OpenGL ES 2.0 the precision level of our data types. As you can imagine, the higher the precision, the more accurate the data type is. But, lower precision has the benefit of being faster. We can set it to lowp, mediump, or highp. There’s a post here that talks more about precision and what the actual constraints are here.
A shader always has the main() function. Our normCoord variable is the location on our incoming texture that we want to use as the color for this fragment. We’ll manipulate this value in order to pixellate from a position in a polar coordinate system.
The first thing we’ll do is convert our coordinate system into a polar coordinate system. The textureCoordinate ranges from 0.0, 0.0 to 1.0, 1.0. The center uniform variable has the same range. In order to describe our screen in polar coordinates, we need a -1.0, -1.0 to 1.0, 1.0 coordinate. The first two lines convert into those values. The third line subtracts the center from normCoord. This effective makes all our calculations originate at the center value we have set, instead of the actual center of our coordinate system.
From here we’ll convert from a cartesian to a polar coordinate system. Once we’re in polar coordinate, we’ll run the same calculation we’d run on a regular pixellate filter, divide by the step size (pixelSize) and subtract the remainder. This has the effect of taking the value every step size instead.
Once we’ve done that we convert the value back into a cartesian coordinate system and add our center value back in. Then we return it to the 0.0, 0.0 1.0, 1.0 range we need for the texture lookup. The next line uses this new normCoord in the texture2D() call. This function is a texture lookup function. It takes two parameters, the two dimensional texture object (inputImageTexture in our case) and the coordinate (textureCoordinateToUse).
Finally, we run the mod operation on the color value. We are using a hard coded step value of 0.1. This will reduce the color range for the red, green, blue (and alpha, but alpha is always 1.0 in our case so . . . ) from 256 steps for each component (256 cubed or 16.8 M colors) to 10 step (1,000 colors).
I feel that my explanation of how and why we move between cartesian and polar coordinates is a little weak. For a more detailed description of using the polar coordinate system in glsl, there’s a great post (now available on wayback machine) here (that’s where I got a bunch of this code from).
That’s our fragment shader. It will run very fast. If we were to do the same operation on the CPU it would take much, much longer to accomplish the same goal. In many cases, the GPU can do live video filtering that wouldn’t be possible on the phone’s CPU.
Finishing the rest of the Class
Once we’ve written the shader, the only other pieces we are required to include are the initialization and the setters for the uniform variables. In the initialization we will set the shader string, set the uniform pointers, and set up any default values.
On to the rest of the code. Add the following after the @implementation:
@synthesize center = _center;
@synthesize pixelSize = _pixelSize;
#pragma mark -
#pragma mark Initialization and teardown
- (id)init;
{
if (!(self = [super initWithFragmentShaderFromString:kGPUImagePolarPixellatePosterizeFragmentShaderString]))
{
return nil;
}
pixelSizeUniform = [filterProgram uniformIndex:@"pixelSize"];
centerUniform = [filterProgram uniformIndex:@"center"];
self.pixelSize = CGSizeMake(0.05, 0.05);
self.center = CGPointMake(0.5, 0.5);
return self;
}
Initializing the class is just a matter of passing in our shader string, and then setting up any and all uniforms. The call to initWithFragmentShaderFromString: passes our shader into the appropriate methods to check and compile it so that it’s ready to run on the GPU. If we wanted to also supply vertex shader (different than the default) there’s also a call to do that.
We have to call [filterProgram uniformIndex:] for each uniform we supply in our class. This sets the handle in the shader to the supplied string and it set the GLint uniform pointer to the appropriate place so the data is passed in.
Finally, we set some default values in the init so that our filter will work, even without the filter’s user needing to supply some values.
The last thing we need to do is set up our getters and setters for the uniform values:
These two setters are very similar to each other, I’ll explain them as though they are the same.
Our setter first sets the class variable to the supplied value. It then calls [GPUImageOpenGLESContext useImageProcessingContext] and [filterProgram use], these two calls set the state of OpenGL so that the supplied data goes to the right program. Remember that the key to the framework is that the filters can be chained together. Those two calls make sure that we have the right shader program loaded when we pass in the data.
Next we set up a float array to pass into the call. The glUniform2fv is one of many similar calls that passes data of a certain type into our shader program. This call takes three arguments, the first is the pointer to the location where the data will live (allowing us to access it by its handle). The second is the count (how many we’re passing). You may think that it should be two because we are passing two floats, however, the call is glUniform2fv, instead of glUniform1fv, the call assumes that we are sending two floats worth of data, so if we passed in 2 for the middle parameter, it would expect two, two float long worth of data. The final parameter is the pointer to the data we are passing.
Structure of GPUImage
GPUImage has classes for inputs (GPUImageVideoCamera, GPUImageMovie, GPUImagePicture), Filters, and outputs (GPUImageView, GPUImageMovieWriter, GPUImageRawData). You chain inputs through one or more filters and then to the output. Each connection is made by adding the next link in the chain as a target to the current link. So, a series would look like, create a camera, filter, and view. Then add the filter as a target to the camera and add the view as a target to the filter. This creates a chain that pulls the framebuffer contents (the output of the OpenGL operations) through the chain to the final output.
One object can have multiple inputs and multiple outputs. Some filters have multiple inputImageTextures, indeed, there is a built in inputImageTexture2 handle that will be available if you set a filter as the target of multiple sources. For example, the GPUImageChromaKeyFilter would be set as the target for both a GPUImagePicture and GPUImageVideoCamera source. This filter replaces any green within a certain range in the video with the contents of the supplied picture.
You can send the output of one filter to several views, or to a view and to a movie writer or to another filter, which sends it to another view. All these structures have examples in the sample code showing you how it’s done. It’s a very flexible structure.
Setting up the View Controller Class
Now that we have a working video, we’ll set up a simple application to use it. Let’s go to the ViewController class that came with the view template we set up earlier. Change the following lines in the implementation file:
We set up a couple of private instance variables, the GPUImageVideoCamera and the GPUImagePolarPixellatePosterizeFilter.
Then we create a chain of filters. We set up the video camera, with the appropriate resolution preset and device position. Then we create a rotation filter because the orientation of the video coming isn’t correct when we’re looking at the phone in portrait mode. We then chain the filters thusly, video camera – rotation filter – polarpixellateposterizefilter – and finally the GPUImageView that we’ll use to display the contents on the screen.
Finally, we set the view of the current view controller class to the GPUImageView. In order for this to work we’ll need to set up the view in the xib (storyboard actually). Go and do that now.
At this point we can run it and we’ll have a working live video filter. But, it might be fun to add some interactivity. The touchesmoved and touchesbegan methods use the location of the touch to set pixelSize uniform variable in our shader. The smallest pixels are acheived by a touch on the top left and the larges pixels by a touch on the bottom right. You can experiment with this filter to get very different results.
Congratulations, you’ve written your first shader.
Other Examples of Image Processing with Shaders
In order to give you some ideas of what else you might do, here are a handful of examples that perform different operations on an incoming image:
Reduce the red and green levels and increase the blue:
lowp vec4 color = sampler2D(inputImageTexture, textureCoordinate);
lowp vec4 alter = vec4(0.1, 0.5, 1.5, 1.0);
gl_FragColor = color * alter;
You can take a look at the actual filters for a bunch more examples, including edge detection, fisheye distortion, and a ton of other cool things. You can get the sample project here.
My book can now be purchased on Amazon. Vol1 and Vol2. My chapters on Core Image and Turn Based Gaming are both in the second Volume. But, if you’re a programmer, the thing is full of awesome. I refer to it every time I need to break out ARC or Storyboards or Twitter integration.
I’ve been working some contract gig for the last six weeks, I’ve been painfully neglectful of both twitter and the blog. I checked my traffic and it’s held steady in spite of my neglect, lucky me.
I’ve got some things on my radar, though. I’ve been working on learning and adding to the GPUImage framework that Brad Larson has put together. It is awesome! If you folks want some tutorials on that I’d be inclined to write some things. Basically, you can write your own shaders to to real time video processing (or stills). If you haven’t yet, download it, install the sample called Filter Showcase and see how awesome it is.
Also, I’m working on the Platformer Starter Kit that will be released on raywenderlich.com. This should be a lot of fun and it will help you go from 0 to fully featured platformer. We’ve engaged a pixel artist and I’m finishing up the programming. I still have to write it, so it may be another month or so, but I’ve been working on this project now since November (in between contracting to pay the bills), so it’s very exciting to see the end of the tunnel.
Today I had the opportunity to talk to the guys from Kodiakreporting.com. The mission of Third Rail Games is to create fun learning experiences. We’ve heard from lots of teachers lately that our game, Factor Samurai, is being used in the classroom. We think that is awesome. But, one thing that we know teachers need is information about what kids are learning when they use educational apps. They need data they can track to evaluate how well these apps teach the core principles to their students. That’s what Kodiak does. We are very excited about what this will do for the educational app (and in our case games) space. If you have any interest in education, check them out.
My latest attempt is to extend the Cocos2d particle system to include a variety of things, multiple particle sprites, animated sprites, flow fields, and eventually a boids algorithm implementation. Today I have a work in progress class that’s by no means ready for any kind of production. I though I’d share it anyway.
In it’s current form, it’s a class that uses a Texture Packer plist and spritesheet to import an animation into a particle system structure. It also uses a flow field to apply different velocities based on position. I’m trying to put together some environmental effects for a game I’m working on. I’d like to get some interesting birds and leaves flying around.
Here’s what it looks like:
It’s running a little slowly on the simulator, but on my iPad 2 it runs at 60fps. The system is defined as having 5000 particles, but I don’t think it actually has that many on screen at a time.
You may think that the particles are just being rotated and scaled, but they are actually distinct frames to give the appearance that the leaves are twirling around in the wind.
It also changes the flow field based on the angle of a dragged touch. You can see the particle movement changing as I drag the mouse cursor around.
I’m working on a subclass of CCParticleSystemQuad that will incorporate this functionality as well as my previous post on particle systems, but that’s not ready yet. You can download the hacked together code here if you are interested.
It’s full of bugs now, so no complaining . . . or just be ok if I ignore any complaining. For example, the touch input assumes that the orientation is portrait, so even though the view will rotate, the touch input in landscape affects the wrong cell.
Basically, the particle struct needs a couple new values, like the current rect index and the time til we switch to the next frame. That variable gets checked and updated each time step, until it’s time to rotate to the next animation frame when it’s reset.
The system class has a new array of CGRects that define the individual frames from the spritesheet. This is populated from the Texture Packer plist.
I also create an array of values that represents the flow field. A flow field is a grid of cells that correspond to a rectangular section on the screen. In the video the flow field is 16 x 16. Each cell has a float that represents an angle in radians. This angle is applied to the velocity of the particles in the cell each time step.
It’s populated by a perlin noise generator to begin with, then a method updates the values based on the touch input.
In all it’s not too complicated. The difficult part is yet to come, incorporating the changes into an elegant subclass of the original CC2D class.
This post is going to be all about openGL. I don’t explain it from a beginner’s perspective, so if you are new the the openGL API, it may not be easy to follow. I have listed what I think are the best resources in the bottom paragraph. If you’re new, definitely read those, especially Jeff Lamarche’s stuff.
I haven’t written anything here for a little while. The main reason is that my attention has turned toward learning openGL and I haven’t gotten to the point that I have anything worth showing off for a little bit.
I remember telling a friend that I’d never want or need to learn openGL because I didn’t plan on writing a 3D game engine any time soon. I figured if I ever wanted to dive into a 3D game, I’d just use Unity or some other commercial engine. Well, that was naive. I have learned that openGL can be used for lots of things that I would want to do.
This first dawned on me when I realized that some of what I was trying to do with Core Image couldn’t perform as well as I wanted. I wanted to be able to process incoming video in more creative ways than what was available with the current set of Core Image filters. I also started to realize that shaders opened up some other interesting opportunities for interesting visuals in Cocos2D 2.0. Finally, I wanted to be able to do some GPGPU stuff. All of those reasons led me to decide to crack the hood on openGL.
So, I looked around a bit for some good places to learn. There doesn’t seem to be a great deal of help for openGL beginner’s out there. It’s not the most approachable API. I did find a few things, but what I wanted to do I couldn’t find sample code for.
It appeared that I needed to learn how to set up openGL to render to a texture so that I could do multipass rendering. This was necessary for post processing effects as well as any kind of GPU based data processing that I want to do.
After some googling I didn’t find much. I did however find this article implementing the game of life doing the processing on the GPU. A week or two later, I found this article with a working sample based on the previous post.
Using that along with the camera class from Brad Larson’s video processing code I came up with a simple version of my Game of Life camera painter (the one I did in Core Image a few months back.) It’s the same kind of thing, it takes a camera frame and masks it using a game of life model. This one can handle cells as small as a single pixel, the previous version started to slow down around 50 x 50 on an iPad 2.
I can’t really go through the code line by line, because I’m still trying to grasp all the openGL concepts myself, but I’ll run through it at a higher level.
When you use openGL you create a render target, this can be on screen or off. If it’s offscreen, you can set that up as a texture to be used as an input for a subsequent render step. If we want to do something like apply a post processing effect, we render first to a target texture, then pass that texture in to a second pass. If we want to draw on top of the current frame, we would use a similar strategy.
As I understand it, we can’t use the same texture target as an input and an output in the same pass. For this reason, we will create two textures, and ping-pong between them each pass.
I’m first gonna show the code that sets up the textures and frame buffers. The code that sets up the layer and the EAGL context is pretty standard, so I won’t cover that. If none of this makes any sense, you should go learn some of the basic concepts for openGL. See the last paragraph.
// create Texture A
glEnable(GL_TEXTURE_2D);
glGenTextures(1, &textureA);
glBindTexture(GL_TEXTURE_2D, textureA);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST); // was GL_LINEAR
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, WIDTH, HEIGHT, 0, GL_RGBA, GL_UNSIGNED_BYTE, NULL);
// create texture B
glEnable(GL_TEXTURE_2D);
glGenTextures(1, &textureB);
glBindTexture(GL_TEXTURE_2D, textureB);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST); // was GL_LINEAR
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
// init texture B with some content...
GLubyte* data = (GLubyte *) malloc(WIDTH*HEIGHT*4*sizeof(GLubyte));
GLubyte val;
for (int i = 0; i < WIDTH * HEIGHT * 4; i+=4) {
if ((arc4random() % 100) > 50) {
val = 0;
} else {
val = 255;
}
data[i] = data[i+1] = data[i+2] = val;
data[i+3] = 255;
}
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, WIDTH, HEIGHT, 0, GL_RGBA, GL_UNSIGNED_BYTE, data);
free(data);
//create fboA and attach texture A to it
glGenFramebuffers(1, &fboA);
glBindFramebuffer(GL_FRAMEBUFFER, fboA);
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, textureA, 0);
//create fboB and attach texture B to it
glGenFramebuffers(1, &fboB);
glBindFramebuffer(GL_FRAMEBUFFER, fboB);
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, textureB, 0);
glGenRenderbuffers(1, &_colorRenderBuffer);
glBindRenderbuffer(GL_RENDERBUFFER, _colorRenderBuffer);
[_context renderbufferStorage:GL_RENDERBUFFER fromDrawable:_eaglLayer];
glGenFramebuffers(1, &framebuffer);
glBindFramebuffer(GL_FRAMEBUFFER, framebuffer);
glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0,
GL_RENDERBUFFER, _colorRenderBuffer);
In the first part we create the textures and the frame buffers and attach the textures to the frame buffers. The last six lines create the render buffer, allocate the storage, and attach the render buffer to a created frame buffer object. This is the recommended way to create a frame buffer for rendering to the screen.
I’m also not going to go over the code that compiles, links, and sets up the shader programs. This is also pretty boilerplate code. Keep in mind that we have two sets of shaders (the vertex program is the same in both set, but it still gets run each pass). We also have two functions that compile the shaders, one for each set. Each set gives us the handle that we use to set that shader in the glUseProgram() call.
Next we’ll look at the render loop. In the original code, a CADisplayLink is set up to call the render loop, but I changed that so it’s just called whenever there’s a new pixelBuffer available from the camera.
Here’s the render loop:
- (void)renderNow {
if (pingpong_counter % 2 == 0) {
[self renderGOLTexture:programHandle texture:textureB frameBuffer:fboA];
} else if (pingpong_counter % 2 == 1) {
[self renderGOLTexture:programHandle texture:textureA frameBuffer:fboB];
}
glBindFramebuffer(GL_FRAMEBUFFER, framebuffer);
glViewport(0, 0, self.frame.size.width, self.frame.size.height);
if (pingpong_counter % 2 == 0) {
[self drawOutputTextureToScreen:texProgramHandle texID:textureB texID2:textureA];
} else {
[self drawOutputTextureToScreen:texProgramHandle texID:textureA texID2:textureB];
}
//And then to actually "blit" that buffer to the screen
//you call "presentRenderbuffer" like so:
glBindRenderbuffer(GL_RENDERBUFFER, _colorRenderBuffer);
[_context presentRenderbuffer:GL_RENDERBUFFER];
//increment the counter so that frame swaps
//the input texture with the output texture
pingpong_counter++;
}
The first if statement checks our counter variable (it just serves as a way to ping pong between the two textures. Based on the current count it calls the renderGOLTexture function passing in one of the sets of textures and appropriate frame buffer. In one case we’re using our textureA rendering to fboB and vice-versa in the other case. Again, we do this because we can’t read from and write to the same openGL object.
Then we switch to our ‘frameBuffer’ object, this is what we’ll use to present to the screen. Once we’re writing to that object we check the counter again and we pass the two textures into the function. We’ll be using both the latest and the previous frame in our program in our final shader. We’ll represent new cells in the game of life as white pixels. Cells that have just died will be represented as green pixels. Dead cells will be represented as black pixels (almost black).
Once our ‘frameBuffer’ object has been written to, we can present the data to the screen which is done by binding the _colorRenderBuffer and calling presentRenderBuffer on the context. Finally, we update the counter for the next pass.
That’s the basic order. First, we call the pass that updates the texture that represents the state of the GOL model. Second, we render that texture to the screen. Finally, present the buffer on the screen and update the counter.
Each pass is using its own shader program. Let’s take a look at the shader that updates the GOL model. Here’s the fragment shader, Automata.glsl:
varying vec2 v_texCoord;
uniform sampler2D tex; //the input texture
uniform float du; //the width of the cells
uniform float dv; //the height of the cells
/*
Any live cell with fewer than two live neighbours dies,
as if caused by under-population.
Any live cell with two or three live neighbours
lives on to the next generation.
Any live cell with more than three live neighbours dies,
as if by overcrowding.
Any dead cell with exactly three live neighbours
becomes a live cell, as if by reproduction.
*/
void main() {
int count = 0;
vec4 C = texture2D( tex, v_texCoord );
vec4 E = texture2D( tex, vec2(v_texCoord.x + du, v_texCoord.y) );
vec4 N = texture2D( tex, vec2(v_texCoord.x, v_texCoord.y + dv) );
vec4 W = texture2D( tex, vec2(v_texCoord.x - du, v_texCoord.y) );
vec4 S = texture2D( tex, vec2(v_texCoord.x, v_texCoord.y - dv) );
vec4 NE = texture2D( tex, vec2(v_texCoord.x + du, v_texCoord.y + dv) );
vec4 NW = texture2D( tex, vec2(v_texCoord.x - du, v_texCoord.y + dv) );
vec4 SE = texture2D( tex, vec2(v_texCoord.x + du, v_texCoord.y - dv) );
vec4 SW = texture2D( tex, vec2(v_texCoord.x - du, v_texCoord.y - dv) );
if (E.r == 1.0) { count++; }
if (N.r == 1.0) { count++; }
if (W.r == 1.0) { count++; }
if (S.r == 1.0) { count++; }
if (NE.r == 1.0) { count++; }
if (NW.r == 1.0) { count++; }
if (SE.r == 1.0) { count++; }
if (SW.r == 1.0) { count++; }
if ( (C.r == 0.0 && count == 3) ||
(C.r == 1.0 && (count == 2 || count == 3))) {
gl_FragColor = vec4(1.0, 1.0, 1.0, 1.0); //cell lives...
} else {
gl_FragColor = vec4(0.0, 0.0, 0.0, 1.0); //cell dies...
}
}
I’m not going to go into too much detail about how glsl works here, but I’ll give you a couple pointers if you’ve never seen this before. It’s something like C. The shader gets run for every fragment. The initial declarations of ‘uniform sampler2D tex’ is the passed in texture (previous frame of the GOL texture) and the ‘varying vec2 v_texCoord’ is the current coordinate for this fragment. ‘dv’ and ‘du’ are the width and height of the GOL model. ‘Uniforms’ are passed in by us and the ‘varying’ is generated by the pipeline (basically).
The texture2D() function is a texture lookup that takes two arguments, the texture and the coordinate. We are getting the current coordinate, plus all the neighboring coordinates and adding up the number of cells that have 1.0 for the value of red (alive). We then just apply the logic of the GOL to see if the current cell is dead or alive. Vec4() creates a vector with 4 elements (red, green, blue, alpha). We use that function to set the current color to either black or white. In a fragment shader we’re always setting gl_FragColor (the current color of the fragment/pixel).
This is the first shader program. It gets run on the first pass, the first half of each render loop. What’s written is the frame buffer that is opposite our input texture. On the next pass the two get flipped and we read from one and write to the other.
A little side note about this structure. In the current implementation, the visual texture represents the game state both logically and visually, but that need not be the case. These two textures that we are ping ponging could contain any array of floats. Each pixel is an array of 4 floats. We could use that data structure to represent anything we wanted and the shader algorithm could process and write it out in any way we wanted. As long as we can convert our data into an array of pixels, we can process it using shaders. This can be very fast.
Back to the code.
So next we’ll look at the function that inputs our uniforms and calls the shader. Here’s the renderGOLTexture function:
-(void) renderGOLTexture:(GLuint) programID texture:(GLuint)texture frameBuffer:(GLuint)frameBuffer {
glBindFramebuffer(GL_FRAMEBUFFER, frameBuffer);
//set the viewport to be the size of the texture
glViewport(0,0, WIDTH, HEIGHT);
//clear the ouput texture
glClear ( GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT );
//bind our automata shader
glUseProgram(programID);
glActiveTexture(GL_TEXTURE0); //make texture register 0 active
glBindTexture(GL_TEXTURE_2D, texture); //bind textureB as our input texture
glUniform1i(glGetUniformLocation(programID, "tex"), 0); //pass texture B as a sampler to the shader
glUniform1f(glGetUniformLocation(programID, "du"), 1.0/WIDTH); //pass in the width of the cells
glUniform1f(glGetUniformLocation(programID, "dv"), 1.0/HEIGHT); //pass in the height of the cells
GLuint _p = glGetAttribLocation(programID, "a_position");
GLuint _t = glGetAttribLocation(programID, "a_texCoord");
// Load the vertex position
glVertexAttribPointer(_p, 3, GL_FLOAT, GL_FALSE, 5 * sizeof(GLfloat), 0);
// Load the texture coordinate
glVertexAttribPointer(_t, 2, GL_FLOAT, GL_FALSE, 5 * sizeof(GLfloat), (GLvoid*) (sizeof(float) * 3));
glEnableVertexAttribArray(_p);
glEnableVertexAttribArray(_t);
glDrawElements(GL_TRIANGLES, sizeof(Indices)/sizeof(Indices[0]), GL_UNSIGNED_BYTE, 0);
glUseProgram(0); //unbind the shader
glBindFramebuffer(GL_FRAMEBUFFER, 0); //unbind the FBO
}
This function sets up the rendering by binding the appropriate frame buffer. It sets the viewport (the number of pixels wide and high, logically, it will fill the screen no matter what the values are). The constants for this are set up at in the header file. I’ve had success with very large game canvases, 960 high by 640 wide on a retina with no problem.
Next, it clears the previous contents of the buffer and set the correct program handle to use. Then we set our incoming texture object and bind it to the correct slot in the openGL environment. We then call glUniform1i(glGetUniformLocation(programID, “tex”), 0). This is how we get our texture data available within the shader.
Next we pass in the other uniforms, dv and du. After that, a very important step, we set up and pass in our vertex and texture coordinate data. This data is stored in a struct that’s set up at the top of the file. Again, I’m not going to cover all these calls here, but I’ll give some references that can explain them at the end of the article.
The call to glDrawElements is what actually does the drawing to the frame buffer. At the end we unbind the framebuffer and the glprogram.
Once this first pass is done, we have a new texture/frame buffer that contains the current state of the GOL. The next thing to do is to render it to the screen frame buffer. We’ll take a look at that shader next. Here’s the Texture.glsl, it’s another fragment shader (I’m not covering the vertex shaders, as all they do is pass the information through to the fragment shader in this code):
This shader has three textures, tex and tex2, which are the current and previous textures generated by the GOL and cam, which is the incoming camera frame. We’re getting the current pixel color from each texture. We then do some tests to see what the state of the pixel is.
If the pixel (fragment) from the current GOL texture is fully red, that means the cell is alive and we just want to use the color from the incoming camera frame. If the current cell is dead, but the cell in the previous frame was alive, we want to use the camera color, but muted, so we’re dividing each color element by 4.0. These colors are vec4, so if we divide the whole structure by a value, it divides each element by that value, neo style. If the current and previous pixels were both dead, then we just want a tiny bit of color from the camera to come through, so we divide by 10.0.
Alright, we can take a look at how we get all three textures into the shader and we’ll be nearly done.
Here’s the drawOutputTextureToScreen function:
- (void) drawOutputTextureToScreen:(GLuint) programID texID:(GLuint) texID texID2:(GLuint) texID2 {
glClearColor(0.0, 0.0, 0.0, 1.0);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
glUseProgram(programID);
glActiveTexture(GL_TEXTURE0); //make texture register 0 active
glBindTexture(GL_TEXTURE_2D, texID); //bind either textureA or textureB
glActiveTexture(GL_TEXTURE1); //make texture register 0 active
glBindTexture(GL_TEXTURE_2D, _cameraTexture);
glActiveTexture(GL_TEXTURE2); //make texture register 0 active
glBindTexture(GL_TEXTURE_2D, texID2); //bind either textureA or textureB
glUniform1i(glGetUniformLocation(programID, "tex"), 0);
glUniform1i(glGetUniformLocation(programID, "cam"), 1);
glUniform1i(glGetUniformLocation(programID, "tex2"), 2);
//assuming you have the dataArray of vertices and texCoords set up...
GLuint _p = glGetAttribLocation(programID, "a_position");
GLuint _t = glGetAttribLocation(programID, "a_texCoord");
glVertexAttribPointer ( _p, 3, GL_FLOAT, GL_FALSE, 5 * sizeof(GLfloat), 0 );
glVertexAttribPointer ( _t, 2, GL_FLOAT, GL_FALSE, 5 * sizeof(GLfloat), (GLvoid*) (sizeof(float) * 3) );
glEnableVertexAttribArray ( _p );
glEnableVertexAttribArray ( _t );
glDrawElements(GL_TRIANGLES, sizeof(Indices)/sizeof(Indices[0]), GL_UNSIGNED_BYTE, 0);
glUseProgram(0);
}
Most of the code here is very similar. We’re not binding the frame buffer here in the function because, if you recall, we do it in the parent function just before we call it.
The only difference is how we go about getting all three textures in to the shader. We can use up to eight textures in our shader. We call glActiveTexture for each of the first three slots and then bind the appropriate handle to the texture. Then in the call to glUniform1i(glGetUniformLocation(programID, “tex”), 0) we also set the second argument to match the slot we picked earlier. That’s all about it. The rest of this function mirrors the other one.
One last thing, lets talk about the camera code. The ColorTrackingCamera class is by Brad Larson and is from his ColorTracking project you can get here. The class sets up a capture session exactly as we did in the Core Image code in other posts on this site.
We set our OpenGLView class as the delegate for the capture session and here’s what we do with the pixel buffer data to get it into our texture object:
- (void)processNewCameraFrame:(CVImageBufferRef)cameraFrame;
{
CVPixelBufferLockBaseAddress(cameraFrame, 0);
int bufferHeight = CVPixelBufferGetHeight(cameraFrame);
int bufferWidth = CVPixelBufferGetWidth(cameraFrame);
// Create a new texture from the camera frame data, display that using the shaders
glGenTextures(1, &_cameraTexture);
glBindTexture(GL_TEXTURE_2D, _cameraTexture);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
// This is necessary for non-power-of-two textures
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
// Using BGRA extension to pull in video frame data directly
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, bufferWidth, bufferHeight, 0, GL_BGRA, GL_UNSIGNED_BYTE, CVPixelBufferGetBaseAddress(cameraFrame));
[self renderNow];
glDeleteTextures(1, &_cameraTexture);
CVPixelBufferUnlockBaseAddress(cameraFrame, 0);
}
We set up a texture the same way we did in the beginning of our program. The glTexImage2D call takes the data from the pixel buffer and puts it into a texture that openGL can use.
We also call the renderNow call from this callback so that we get a render loop each time we get a new camera frame. We don’t need to update the game any more often than that and it appears to give about 30 frames per second on an iPhone 4S or iPad 2. I’m not sure of the performance on a previous gen device, but I suspect it would do fine on an iPhone 4.
That’s about all for this post. I realize that I didn’t explain many of the basics of openGL, so if you are new to the API, here are the places I used to learn most of the things I didn’t cover here. I’d start with Jeff LaMarche’s openGL resources. These are excellent and easy to follow. Then I’d do in no particular order Ray Wenderlich’s OpenGL ES 2.0 for iPhone parts 1 and 2 and Apple’s OpenGL ES Programming Guide for iOS, especially the part about Drawing as it talks about setting up all these frame buffers. There are also some books out there, Philip Rideout’s iPhone 3D Programming, though this book is a C++ book, which I’m useless with. Also, this book, also C++, has some good things in it as well. I think the blog posts are easier to follow and get me to and understanding quicker (than the books), but that could just be me.
I’d like to write a bunch more stuff about openGL. If this is useful let me know in the comments. It will help me learn and I think there’s a lot of cool things to be done with shaders in Cocos2D. Bump mapping, lighting, cool pixellation shader, edge detection, gpu based particle engines, maybe come gpu physics, I don’t really know, but I think it could be cool. If anyone has any suggestions about good openGL resources let me know that, too.
For the first time I can review my new year’s resolutions (for this year). At the beginning of the year, among other things I had the following goal: Lose 40 lbs, get down to about 220.
The year went along with minor attempts at this goal, but in September something happened that helped me get serious. We (my wife and some other family) were bored and so we watched this semi-documentary/advertisement called, “Fat, sick, and nearly dead”. It’s about this Austrailian guy who goes on a vegetable juice only diet and loses 80 lbs in about 3 months.
So, we decided to try it. We got the $400 juicer that he uses in the show (we were serious) and started juicing. I started off eating a variety of vegetable and fruits and juicing, but I wasn’t 100%. In spite of that I lost about 10 pounds in two weeks. We committed ourselves to a two week trial of this diet. I have to say that the first five days I felt like I was starving, I had headaches, I was tired and irritable all the time. The first five days were painful.
Then it started to get better, and I started to lose. The other people that started the diet with me all had better results in those first two weeks than I did. My dad, who is a little smaller than me, lost 16 pounds.
So, after the first two weeks and a little break, I started again, with more commitment. I went full juice for the next two weeks. During this time I also went back to the gym and was doing weight lifting circuits, three times a week. I did this on two weeks, off for a few days three more times. By mid December I got down to 225 pounds. Which represented a 40 pound weight loss and my target goal.
Here’s my before and after.
I really feel good about what I was able to accomplish this year. I have had back problems (I sit a lot) and have for about three years. It’s much better now. I think a little more exercise and a few more pounds and I may able to forget that I ever had back pain.
In a previous life (pre-marriage) I was a personal trainer at a local gym. I was never super ripped, but I was in pretty good shape and I felt pretty good about how much weight I could lift. So looking the mirror for the last few years has been an exercise in shame. No more.
On to this year’s resolutions. I’d like to lose a few more pounds and put on some muscle. I’d like to be about 10% bodyfat. I’m not sure what I am now, I’ll have to go get tested. My guess is about 20%.
This year I’m gonna try some new things, the first is self tracking. I bought both a BodyMedia arm band and a fitbit. I’m going to play with both of them and see how they work. I’m a data geek and have some experience with the science behind weight loss and muscle building, so being able to track activity and calorie expenditure with these devices gets me going.
The other big benefit with both of these devices (and why I chose them over all the other available devices) is that they both have an API which can be used to build third party apps. I’m interested in putting something together after I’ve played with both and looked at what data is available. I’d thinking something that’s a cross between Fleetly and Epic Win.
Here’s my first impression of the two devices:
BodyMedia v Fitbit
BodyMedia is supposedly more accurate, and I believe that claim, because it is measuring things by making contact with your skin. The fitbit is essentially a pedometer. If I am lifting weights how does the fitbit even know that I’m expelling extra calories? At some point I’ll use both simultaneously to see what the data coming out of each looks like.
The fitbit is easier, it syncs wirelessly with it’s dock, the bodymedia requires that I plug it in. The fitbit can be placed anywhere, in a pocket, the bodymedia needs to be on my arm.
Both have iphone apps, but as far as I can tell so far, neither push data (from the device) to the central server. The fitbit app looks like it’s an extension of the website (you can log food, activities, etc), but it can’t read data off the device (even though the syncing with the dock is bluetooth, which the iphone can do). The bodymedia app pulls data off the device for realtime info, but it doesn’t push that information to the server.
Both have websites that record food intake, activity logging, etc. I haven’t dug in enough to know which I prefer. Both have APIs, but I’ll need to explore that more thoroughly to know the limitations of each.
The fitbit has a sexier design:
On each you still have to record your food and water intake manually (obviously). This process of recording is probably the hardest thing about dieting. You get tired of always stopping and writing things down. It can be a lot of work. The benefit of writing it all down is that it forces you to examine your behavior. We do a lot of things on auto-pilot, or we tell white lies to ourselves about how much we actually eat. It’s hard to eat a whole package of cookies when you know that you have to write down that you did it. I kind of think that just writing down what you eat is enough for most people to lose weight, because when you pay close attention to your behavior, that’s often enough incentive to make changes.
I’ll write again when I’ve played with each system a bit more.
I’ll write some more about the merits of each when I’ve used them both for a few weeks. I’m pretty excited to see how tracking data
Most of my development work is centered around educational games. I am co-founder of Third Rail Games, LLC. A while back I wanted to modify the particle system in Cocos2D so that it would put out streams of letters in a given sequence instead of the one sprite.
I’ll explain the simple modification required to create a particle system that puts out a string of letters, and you’ll see that the same logic could be used to create a particle system that pulls from a sprite sheet.
Here’s the code from the class. I’ll explain what each element does.
This new class is a subclass of the CCParticleSystemQuad class in Cocos2d. Besides the cocos2d and foundation classes, we’re importing a class extension file, CCBMFontConfiguration+atlasName.h. This extension exposes the altasName property of the Bitmap font classes we’ll use. It allows us to get the name of the texture file from a bitmap font class. I used Glyph Designer to create my bitmap font file, but you can use any bitmap font generator tool for this.
We’re adding two instance variables. The first stores the bitmap font configuration information which we’ll use to lookup the rect for each letter. The second stores our string that will be used as the list of letters to use for the particle generator. The particle generator will create the particles in the order of the letters in the string.
If you look at the implementation file, the first two methods are the getter and setter for the particleString property. The reason I created my own is that I need the custom setter to call the initTexCoordsWithRect: method every time the string is changed.
Next is our initialization method, we pass in all the information we’ll need to set up the particle generator. The font config file contains all the information about each letter in the bitmap, we’ll use this to lookup the rect for each letter. The plist file is the same that one would use to set up a CCParticleSystemQuad in Cocos2d. Finally, we pass in the string that we’ll use to create the letter sequence.
First the method sets the fntConfig variable using the input file. We then use this object and the class extension method to get the name of the atlasName property (name of the texture file) and set it to the texture of the particle generator. Then we set the particleString, which also calls the initTexCoordsWithRect: method.
The next method is the getLetterRect:(int)position method. We use the position of the particle in the particles array to lookup what letter it should be. This method seems to work, even though it’s not obvious why a particles position in the array should stay constant. I’ll have to investigate this further, but for now it works fine.
The particle system just starts the string over after having output all the letters. We use a modulo to get the position in the string if the array position is greater than the string length. Using the position we get the character (unichar) from the string. The CCBMFontConfiguration object has a struct that contains most of the information about the bitmap font, including the rect. The unichar character also serves as the index for the associated data in the struct that contains our rect info. We use this class and struct to lookup the rect.
The last method is the initTexCoordsWithRect. This method exists in the CCParticleSystemQuad class and normally it’s job is to set the texture coordinates for all the particles. Usually it’s the same rect for every particle and so the system passes in the rect. But, in our case we’ll just pass in CGRectZero and return a different rect for each particle.
The first thing we need to do is test whether we’ve set the particleString class. If this is nil it will crash, so we just abandon the method at that point. If the string is set, then we can proceed.
Most of this code is from the CCParticleSystem class. Cocos2d uses a structure called quads_ to hold the texture information it passes to Cocos2d. We’re assigning this data structure the values from the getLetterRect method. This will use a different section of our texture for each particle.
You can see that you could use the same logic with a texture atlas to create a particle system that uses different images. You’d just need to set the texture to your sprite sheet and change the getLetterRect method to return a rect from that sprite sheet.
Here’s a sample with all the code. It’s using Cocos2d 2.0, so you may need to modify one of the calls in order for it to work with Cocos2D 1.0, the line that sets the texture in the init method.
This isn’t a very robust class as I just took it this far, it probably needs some more testing, but I figured I’d put it out there for any that may benefit from it. Enjoy.
My very first idea for an iphone app was something that would allow me to navigate the language used in tweets. I wanted to create an app that pulled twitter data, based on a search phrase, parse the words in that request, pull out represent those words by frequency or importance, and present that visually to the user. Then, the user could pick from the resulting words and start the whole process again.
Well I eventually was able to create that application, it’s called TweetViz and it’s available on the store. It just uses a word frequency count to display the most commonly associated words with the search string. However, creating that app was kind of a pain. I had to learn how to use the MGTwitterEngine, which is a great library, but still a bit difficult, there are a bunch of dependencies, not all the functions work together, because some are json only and others are xml. It usually takes me a good several hours to remember how to use the library and set up the dependencies again.
Well all that’s over with the new twitter integration. It’s a more elegant interface to the twitter api and the accounts framework takes care of authentication (although here we will use the search api, which doesn’t require authentication).
In this post I’ll put together in less than 100 (plus libraries and boilerplate stuff) lines of code, a project equal to the one that took me about a week to complete previously. We’ll query twitter, organize the results by frequency count, eliminate the words that don’t have meaning (am, the, a, etc) and display it all in a treemap. I’ll use the NSLinguisticTagger class to group words that have the same meaning (reading, read, reads) together.
Here’s what it will look like:
First, we’re gonna use the treemap visualization library (tweetviz uses a commercial library, so I can’t post it here). You can get the original here, but I’ve made a few tweaks so in order to follow along download this version.
The first thing we need to do is add the twitter library/framework.
Once you’ve done that open up the ViewController file and add the following method (also declare it in the header):
The TWRequest object is created to match the structure of the twitter api. So for each request there are several parts, 1) The url, 2) parameters, 3) The request method.
We just use the url that twitter gives us in the twitter documentation, we don’t have to add the parameter string, the TWRequest will do that for us based on the supplied NSDictionary fed into the parameters argument. In those cases where you have multiple values, like the users/lookup api call, just convert those multiple values into a comma-delimited string.
In the block we use the new NSJSONSerialization object to convert the returned json to an NSDictionary. It’s so much easier . . .
We’re using perry as our search string, depending on your interests you may think I’m looking for Rick Perry or Katy Perry. The results we get indicate that most people would rather talk about Katy than Rick . . . and I guess I don’t blame them. You should get a log output that looks like this:
Next we want to combine all the statuses into one giant status. We also want to strip out some of the stuff we aren’t going to want to look at. Lets remove all the @mentions, the hashtags, the links, and all the stopwords.
Now if you run it, you’ll get one long string with a bunch of the links and other things removed. Like this:
Now it’s time to break out the NSLinguisticTagger class. We’ll be using this to color code our common words, whether they are verbs, nouns, or names of people. We could also use this class to determine whether a phrase is a Named Entity. But we won’t do that here.
Continuing in that same method, add this code (replacing that last NSLog call):
Sorry about how hard that last block is to read. What we’re doing here is first creating a tagger. Our tagger can handle both Lexical Class tagging (verb, noun, particle, etc) and Lemma (word stemming, reading, read, reads, all the same stem). We pass in an array to tell the tagger what kinds of things it should tag.
We’re also giving it some other options, I only half understand them so I’m not going to try to explain them. Read the Class Reference for more info about this. I only want it to look at the words of the text, so I think those two bitmask (no whitespace, no punctuation tagging) options are what I wanted. It works.
Next, we set the string the tagger will be working with. This is essential, if you get a NSRangeException error, then you may have skipped this part.
We are creating a dictionary that we’ll use in a minute, add an NSMutableDictionary called *tags to the header file now.
Finally we tell it to start tagging. We pass in the string range, the scheme for this tagging pass, and some more options.
Finally we give it a block to deal with the tagging process. Inside the block we are going to get a range in the original string that represents the tagged section. We also get, the sentence range, the tag for that range, and then a pointer to a Boolean. If we want the tagging to stop at some point, we’d set that bool to yes in our block.
If you run it now, you’ll see we got tags:
We want our tagger to do two things, the first is to stem the word, so we can count the token (fancy word that means . . . word), and then to tell us what the lexical class of that token is. We’ll be constructing a dictionary (that’s what that tags dictionary is for) that we’ll then use to populate our treemap. So, now we’ll add some more code that will do that double duty tagging and also reorganize the data so it will be easier to use.
The first thing we’re doing is reading in a plist of ‘stop words.’ These are words like be, have, a, to, like etc. You can download that file here or get it at the end of the post with the finished code. Words that are functional in the English language but when we are doing analysis they don’t help us. So we take them out. This list also includes a few things that are common to twitter text, like RT and "e;. Also not helpful if we’re looking to understand associations between our search phrase and the words people use when talking about that topic.
This stopwords list is from my other project, it contains every tense of each verb. This isn’t necessary with the NSLinguisticTagger class, so the list could be reduced to just the basic forms of those verbs, but it’s good enough for now.
Next we invoke the same tagger method. The difference is when we get into the block. The first thing we do is get our second tag, using the tagAtIndex:scheme:tokenRange:sentenceRange: method. We can call this method and give it an index number (representing the position in our main string) and it will give us back the tag with the specified scheme. In this way we can get our second tag scheme inside the block that’s providing the first one. There may be another way to do this, but this is what I came up with so far.
So now we have tag, which tells us the Lemma or stem, and we have the tag2, which is the Lexical Class.
Next we’re going to construct our dictionary. We need to first check to see if the tag is in our stopwords array, this means it’s a word that is probably common, but not meaningful. If it is, we do nothing with it. If it’s not in the stopwords list we also need to make sure it’s not null, because we’ll be using this value as the key to our dictionary and we cannot use a null key.
The next if statement just checks to see if it’s already been added. If not we create the dictionary that contains the Lexical class and a count of one. If the key does already exist, we increment the count by one.
That’s it. We now have a dictionary that contains a frequency count of all the tagged tokens in our twitter results.
Next we need to integrate this data into our treemap UI. We’ll restructure our data and fit it into the existing sortedKeys and data instance variables so we can leave most of the treemap code untouched.
Add this to the bottom of the searchTwitter method after the close of the [tagger enumerateTagsInRange method call:
NSMutableDictionary *mut = [NSMutableDictionary dictionary];
for (NSString *k in tags) {
if ([[[tags objectForKey:k] objectForKey:@"count"] intValue] > 1 && ![k isEqualToString:str]) {
[mut setObject:[[tags objectForKey:k] objectForKey:@"count"] forKey:k];
}
}
data = mut;
sortedKeys = [[data allKeys] sortedArrayUsingSelector:@selector(caseInsensitiveCompare:)];
[(TreemapView *)self.view createNodes];
The createNodes method is private on the TreemapView object, add the method to the header so that we can call it. Also, comment out everything in viewDidLoad except for our call to searchTwitter.
In this method we are iterating through our tags object and including all the tags that have more than a count of one and aren't our search string. We then put the data into the two instance variables. This just makes it easier, so we aren't changing a bunch of code we don't need to. In a proper implementation I'd probably go through and change all the treemap methods to deal my own data structure.
Then we call createNodes on the treemap view. We call this instead of reloadData, because this treemap class calls resizeNodes on the reloadData call. If the new nodes and the old nodes don't have the same number, it will throw an error.
Go ahead and run it now. You should have a treemap populated with your counts and tags from twitter:
Finally, lets change the updateCell method to change the cells color based on what kind of tag it is (noun, verb, etc) and lets change the tappedcell method so that it calls the searchTwitter method. These are delegate methods of the Treemap class and already exist in the file.
First we get the "tense" for each tag from our tags dictionary, then we use that to create a different color algorithm for each kind. Finally, we set the cell background color to our calculation.
We use the index to get the cell. From the cell we can get the tag name for that cell. We pass that into our searchTwitter method and we'll reload a new data set and display it. It takes a little while to reload the data, so it's a little clunky, but works for a demo!
This is what it should look like:
You can download the completed code here. If you're interested in a deeper treatment of either the NSLinguisticTagger or Twitter integration, both are covered in iOS 5 By Tutorials.
I found this comment on stackoverflow, pixelfreak is the author. But, there are multiple places where it is said that you cannot set the -fno-objc-arc flag on multiple files at once in Xcode. Not true. You can read it from the source here.
But, here’s the deal. I, and doubtless many of you, use third party libraries like Sharekit, for example, that aren’t compatible with ARC yet. One solution is to set the -fno-objc-arc flag for the files that you don’t want using ARC. Sharekit has a ton of files so it would be garbage to have to go through one at a time.
The solution is to select multiple files, hit enter (not double-click), put in the -fno-objc-arc flag, hit enter again. You’ll see that all your selected files now have the flag set. I was glad to find this piece of information.
