My Quest to Learn OpenGL 3

This post is going to be all about openGL. I don’t explain it from a beginner’s perspective, so if you are new the the openGL API, it may not be easy to follow. I have listed what I think are the best resources in the bottom paragraph. If you’re new, definitely read those, especially Jeff Lamarche’s stuff.

I haven’t written anything here for a little while. The main reason is that my attention has turned toward learning openGL and I haven’t gotten to the point that I have anything worth showing off for a little bit.

I remember telling a friend that I’d never want or need to learn openGL because I didn’t plan on writing a 3D game engine any time soon. I figured if I ever wanted to dive into a 3D game, I’d just use Unity or some other commercial engine. Well, that was naive. I have learned that openGL can be used for lots of things that I would want to do.

This first dawned on me when I realized that some of what I was trying to do with Core Image couldn’t perform as well as I wanted. I wanted to be able to process incoming video in more creative ways than what was available with the current set of Core Image filters. I also started to realize that shaders opened up some other interesting opportunities for interesting visuals in Cocos2D 2.0. Finally, I wanted to be able to do some GPGPU stuff. All of those reasons led me to decide to crack the hood on openGL.

So, I looked around a bit for some good places to learn. There doesn’t seem to be a great deal of help for openGL beginner’s out there. It’s not the most approachable API. I did find a few things, but what I wanted to do I couldn’t find sample code for.

It appeared that I needed to learn how to set up openGL to render to a texture so that I could do multipass rendering. This was necessary for post processing effects as well as any kind of GPU based data processing that I want to do.

After some googling I didn’t find much. I did however find this article implementing the game of life doing the processing on the GPU. A week or two later, I found this article with a working sample based on the previous post.

Using that along with the camera class from Brad Larson’s video processing code I came up with a simple version of my Game of Life camera painter (the one I did in Core Image a few months back.) It’s the same kind of thing, it takes a camera frame and masks it using a game of life model. This one can handle cells as small as a single pixel, the previous version started to slow down around 50 x 50 on an iPad 2.

You can download that here..

I can’t really go through the code line by line, because I’m still trying to grasp all the openGL concepts myself, but I’ll run through it at a higher level.

When you use openGL you create a render target, this can be on screen or off. If it’s offscreen, you can set that up as a texture to be used as an input for a subsequent render step. If we want to do something like apply a post processing effect, we render first to a target texture, then pass that texture in to a second pass. If we want to draw on top of the current frame, we would use a similar strategy.

As I understand it, we can’t use the same texture target as an input and an output in the same pass. For this reason, we will create two textures, and ping-pong between them each pass.

I’m first gonna show the code that sets up the textures and frame buffers. The code that sets up the layer and the EAGL context is pretty standard, so I won’t cover that. If none of this makes any sense, you should go learn some of the basic concepts for openGL. See the last paragraph.

// create Texture A       
 
    glEnable(GL_TEXTURE_2D);
	glGenTextures(1, &textureA);
	glBindTexture(GL_TEXTURE_2D, textureA);
	glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST); // was GL_LINEAR
	glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
	glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
	glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, WIDTH, HEIGHT, 0, GL_RGBA, GL_UNSIGNED_BYTE, NULL);	

	// create texture B

	glEnable(GL_TEXTURE_2D);
	glGenTextures(1, &textureB);
	glBindTexture(GL_TEXTURE_2D, textureB);
	glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST); // was GL_LINEAR
	glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
	glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);

        // init texture B with some content...

	GLubyte* data = (GLubyte *) malloc(WIDTH*HEIGHT*4*sizeof(GLubyte));
	GLubyte val;
	for (int i = 0; i < WIDTH * HEIGHT * 4; i+=4) {
		if ((arc4random() % 100) > 50) {
			val = 0;
		} else {
			val = 255;
		}
		data[i] = data[i+1] = data[i+2] = val;
		data[i+3] = 255;
	}
	glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, WIDTH, HEIGHT, 0, GL_RGBA, GL_UNSIGNED_BYTE, data);
	free(data);

        //create fboA and attach texture A to it
	glGenFramebuffers(1, &fboA);
	glBindFramebuffer(GL_FRAMEBUFFER, fboA);
	glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, textureA, 0);

	//create fboB and attach texture B to it
	glGenFramebuffers(1, &fboB);
	glBindFramebuffer(GL_FRAMEBUFFER, fboB);
	glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, textureB, 0);

    glGenRenderbuffers(1, &_colorRenderBuffer);
    glBindRenderbuffer(GL_RENDERBUFFER, _colorRenderBuffer);                
    [_context renderbufferStorage:GL_RENDERBUFFER fromDrawable:_eaglLayer];    

    glGenFramebuffers(1, &framebuffer);
    glBindFramebuffer(GL_FRAMEBUFFER, framebuffer);
    glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, 
							  GL_RENDERBUFFER, _colorRenderBuffer);

In the first part we create the textures and the frame buffers and attach the textures to the frame buffers. The last six lines create the render buffer, allocate the storage, and attach the render buffer to a created frame buffer object. This is the recommended way to create a frame buffer for rendering to the screen.

I’m also not going to go over the code that compiles, links, and sets up the shader programs. This is also pretty boilerplate code. Keep in mind that we have two sets of shaders (the vertex program is the same in both set, but it still gets run each pass). We also have two functions that compile the shaders, one for each set. Each set gives us the handle that we use to set that shader in the glUseProgram() call.

Next we’ll look at the render loop. In the original code, a CADisplayLink is set up to call the render loop, but I changed that so it’s just called whenever there’s a new pixelBuffer available from the camera.

Here’s the render loop:

- (void)renderNow {

	if (pingpong_counter % 2 == 0) {
        [self renderGOLTexture:programHandle texture:textureB frameBuffer:fboA];
	} else if (pingpong_counter % 2 == 1) {
        [self renderGOLTexture:programHandle texture:textureA frameBuffer:fboB];
	}

	glBindFramebuffer(GL_FRAMEBUFFER, framebuffer);
        glViewport(0, 0, self.frame.size.width, self.frame.size.height);
	
	if (pingpong_counter % 2 == 0) {
		[self drawOutputTextureToScreen:texProgramHandle texID:textureB texID2:textureA];
	} else {
		[self drawOutputTextureToScreen:texProgramHandle texID:textureA texID2:textureB];
	}

	//And then to actually "blit" that buffer to the screen
	//you call "presentRenderbuffer" like so:
	
	glBindRenderbuffer(GL_RENDERBUFFER, _colorRenderBuffer);
	[_context presentRenderbuffer:GL_RENDERBUFFER];
	
	//increment the counter so that frame swaps
	//the input texture with the output texture
	pingpong_counter++;

}

The first if statement checks our counter variable (it just serves as a way to ping pong between the two textures. Based on the current count it calls the renderGOLTexture function passing in one of the sets of textures and appropriate frame buffer. In one case we’re using our textureA rendering to fboB and vice-versa in the other case. Again, we do this because we can’t read from and write to the same openGL object.

Then we switch to our ‘frameBuffer’ object, this is what we’ll use to present to the screen. Once we’re writing to that object we check the counter again and we pass the two textures into the function. We’ll be using both the latest and the previous frame in our program in our final shader. We’ll represent new cells in the game of life as white pixels. Cells that have just died will be represented as green pixels. Dead cells will be represented as black pixels (almost black).

Once our ‘frameBuffer’ object has been written to, we can present the data to the screen which is done by binding the _colorRenderBuffer and calling presentRenderBuffer on the context. Finally, we update the counter for the next pass.

That’s the basic order. First, we call the pass that updates the texture that represents the state of the GOL model. Second, we render that texture to the screen. Finally, present the buffer on the screen and update the counter.

Each pass is using its own shader program. Let’s take a look at the shader that updates the GOL model. Here’s the fragment shader, Automata.glsl:


varying vec2 v_texCoord;
uniform sampler2D tex; //the input texture

uniform float du; //the width of the cells
uniform float dv; //the height of the cells

/*
Any live cell with fewer than two live neighbours dies,
as if caused by under-population.
Any live cell with two or three live neighbours
lives on to the next generation.
Any live cell with more than three live neighbours dies,
as if by overcrowding.
Any dead cell with exactly three live neighbours
becomes a live cell, as if by reproduction.
*/

void main() {
	int count = 0;

	vec4 C = texture2D( tex, v_texCoord );
	vec4 E = texture2D( tex, vec2(v_texCoord.x + du, v_texCoord.y) );
	vec4 N = texture2D( tex, vec2(v_texCoord.x, v_texCoord.y + dv) );
	vec4 W = texture2D( tex, vec2(v_texCoord.x - du, v_texCoord.y) );
	vec4 S = texture2D( tex, vec2(v_texCoord.x, v_texCoord.y - dv) );
	vec4 NE = texture2D( tex, vec2(v_texCoord.x + du, v_texCoord.y + dv) );
	vec4 NW = texture2D( tex, vec2(v_texCoord.x - du, v_texCoord.y + dv) );
	vec4 SE = texture2D( tex, vec2(v_texCoord.x + du, v_texCoord.y - dv) );
	vec4 SW = texture2D( tex, vec2(v_texCoord.x - du, v_texCoord.y - dv) );

	if (E.r == 1.0) { count++; }
	if (N.r == 1.0) { count++; }
	if (W.r == 1.0) { count++; }
	if (S.r == 1.0) { count++; }
	if (NE.r == 1.0) { count++; }
	if (NW.r == 1.0) { count++; }
	if (SE.r == 1.0) { count++; }
	if (SW.r == 1.0) { count++; }

	if ( (C.r == 0.0 && count == 3) ||
		(C.r == 1.0 && (count == 2 || count == 3))) {
			gl_FragColor = vec4(1.0, 1.0, 1.0, 1.0); //cell lives...
	} else {
		gl_FragColor = vec4(0.0, 0.0, 0.0, 1.0); //cell dies...
	}
}

I’m not going to go into too much detail about how glsl works here, but I’ll give you a couple pointers if you’ve never seen this before. It’s something like C. The shader gets run for every fragment. The initial declarations of ‘uniform sampler2D tex’ is the passed in texture (previous frame of the GOL texture) and the ‘varying vec2 v_texCoord’ is the current coordinate for this fragment. ‘dv’ and ‘du’ are the width and height of the GOL model. ‘Uniforms’ are passed in by us and the ‘varying’ is generated by the pipeline (basically).

The texture2D() function is a texture lookup that takes two arguments, the texture and the coordinate. We are getting the current coordinate, plus all the neighboring coordinates and adding up the number of cells that have 1.0 for the value of red (alive). We then just apply the logic of the GOL to see if the current cell is dead or alive. Vec4() creates a vector with 4 elements (red, green, blue, alpha). We use that function to set the current color to either black or white. In a fragment shader we’re always setting gl_FragColor (the current color of the fragment/pixel).

This is the first shader program. It gets run on the first pass, the first half of each render loop. What’s written is the frame buffer that is opposite our input texture. On the next pass the two get flipped and we read from one and write to the other.

A little side note about this structure. In the current implementation, the visual texture represents the game state both logically and visually, but that need not be the case. These two textures that we are ping ponging could contain any array of floats. Each pixel is an array of 4 floats. We could use that data structure to represent anything we wanted and the shader algorithm could process and write it out in any way we wanted. As long as we can convert our data into an array of pixels, we can process it using shaders. This can be very fast.

Back to the code.

So next we’ll look at the function that inputs our uniforms and calls the shader. Here’s the renderGOLTexture function:

-(void) renderGOLTexture:(GLuint) programID texture:(GLuint)texture frameBuffer:(GLuint)frameBuffer {
    glBindFramebuffer(GL_FRAMEBUFFER, frameBuffer);
    //set the viewport to be the size of the texture
    glViewport(0,0, WIDTH, HEIGHT);
    
    //clear the ouput texture
    glClear ( GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT );
    
    //bind our automata shader
    glUseProgram(programID);
    
    glActiveTexture(GL_TEXTURE0); //make texture register 0 active
    glBindTexture(GL_TEXTURE_2D, texture); //bind textureB as our input texture
    
    glUniform1i(glGetUniformLocation(programID, "tex"), 0); //pass texture B as a sampler to the shader
    glUniform1f(glGetUniformLocation(programID, "du"), 1.0/WIDTH); //pass in the width of the cells
    glUniform1f(glGetUniformLocation(programID, "dv"), 1.0/HEIGHT); //pass in the height of the cells
    
    GLuint _p = glGetAttribLocation(programID, "a_position");
	GLuint _t = glGetAttribLocation(programID, "a_texCoord");
    // Load the vertex position
    glVertexAttribPointer(_p, 3, GL_FLOAT, GL_FALSE, 5 * sizeof(GLfloat), 0);
    // Load the texture coordinate
    glVertexAttribPointer(_t, 2, GL_FLOAT, GL_FALSE, 5 * sizeof(GLfloat), (GLvoid*) (sizeof(float) * 3));
    
    glEnableVertexAttribArray(_p);
    glEnableVertexAttribArray(_t);
    
    glDrawElements(GL_TRIANGLES, sizeof(Indices)/sizeof(Indices[0]), GL_UNSIGNED_BYTE, 0);
    
    glUseProgram(0); //unbind the shader
    glBindFramebuffer(GL_FRAMEBUFFER, 0); //unbind the FBO
}

This function sets up the rendering by binding the appropriate frame buffer. It sets the viewport (the number of pixels wide and high, logically, it will fill the screen no matter what the values are). The constants for this are set up at in the header file. I’ve had success with very large game canvases, 960 high by 640 wide on a retina with no problem.

Next, it clears the previous contents of the buffer and set the correct program handle to use. Then we set our incoming texture object and bind it to the correct slot in the openGL environment. We then call glUniform1i(glGetUniformLocation(programID, “tex”), 0). This is how we get our texture data available within the shader.

Next we pass in the other uniforms, dv and du. After that, a very important step, we set up and pass in our vertex and texture coordinate data. This data is stored in a struct that’s set up at the top of the file. Again, I’m not going to cover all these calls here, but I’ll give some references that can explain them at the end of the article.

The call to glDrawElements is what actually does the drawing to the frame buffer. At the end we unbind the framebuffer and the glprogram.

Once this first pass is done, we have a new texture/frame buffer that contains the current state of the GOL. The next thing to do is to render it to the screen frame buffer. We’ll take a look at that shader next. Here’s the Texture.glsl, it’s another fragment shader (I’m not covering the vertex shaders, as all they do is pass the information through to the fragment shader in this code):

precision mediump float;
varying vec2 v_texCoord;
uniform sampler2D tex;
uniform sampler2D tex2;
uniform sampler2D cam;

void main() {
    vec4 color, color2, camColor;
    color = texture2D(tex, v_texCoord );
    color2 = texture2D(tex2, v_texCoord );
    camColor = texture2D(cam, v_texCoord );
    
    if (color.r > 0.5) {
        color = camColor;
    } 
    if (color2.r > 0.5 && color.r < 0.5) {
        color = camColor / 4.0;
    }
    if (color2.r < 0.5 && color.r < 0.5) {
        color = camColor / 10.0;
    }
    
    gl_FragColor = color;
}

This shader has three textures, tex and tex2, which are the current and previous textures generated by the GOL and cam, which is the incoming camera frame. We’re getting the current pixel color from each texture. We then do some tests to see what the state of the pixel is.

If the pixel (fragment) from the current GOL texture is fully red, that means the cell is alive and we just want to use the color from the incoming camera frame. If the current cell is dead, but the cell in the previous frame was alive, we want to use the camera color, but muted, so we’re dividing each color element by 4.0. These colors are vec4, so if we divide the whole structure by a value, it divides each element by that value, neo style. If the current and previous pixels were both dead, then we just want a tiny bit of color from the camera to come through, so we divide by 10.0.

Alright, we can take a look at how we get all three textures into the shader and we’ll be nearly done.

Here’s the drawOutputTextureToScreen function:

- (void) drawOutputTextureToScreen:(GLuint) programID texID:(GLuint) texID texID2:(GLuint) texID2 {
	glClearColor(0.0, 0.0, 0.0, 1.0);
	glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
	
	glUseProgram(programID);
	
	glActiveTexture(GL_TEXTURE0); //make texture register 0 active
	glBindTexture(GL_TEXTURE_2D, texID); //bind either textureA or textureB

    glActiveTexture(GL_TEXTURE1); //make texture register 0 active
	glBindTexture(GL_TEXTURE_2D, _cameraTexture);
    
	glActiveTexture(GL_TEXTURE2); //make texture register 0 active
	glBindTexture(GL_TEXTURE_2D, texID2); //bind either textureA or textureB
	
    glUniform1i(glGetUniformLocation(programID, "tex"), 0);
    glUniform1i(glGetUniformLocation(programID, "cam"), 1);
    glUniform1i(glGetUniformLocation(programID, "tex2"), 2);
     
	//assuming you have the dataArray of vertices and texCoords set up...
	GLuint _p = glGetAttribLocation(programID, "a_position");
	GLuint _t = glGetAttribLocation(programID, "a_texCoord");
	glVertexAttribPointer ( _p, 3, GL_FLOAT, GL_FALSE, 5 * sizeof(GLfloat), 0 );
	glVertexAttribPointer ( _t, 2, GL_FLOAT, GL_FALSE, 5 * sizeof(GLfloat), (GLvoid*) (sizeof(float) * 3) );
	
	glEnableVertexAttribArray ( _p );
	glEnableVertexAttribArray ( _t );
	
	glDrawElements(GL_TRIANGLES, sizeof(Indices)/sizeof(Indices[0]), GL_UNSIGNED_BYTE, 0);
	
	glUseProgram(0);
}

Most of the code here is very similar. We’re not binding the frame buffer here in the function because, if you recall, we do it in the parent function just before we call it.

The only difference is how we go about getting all three textures in to the shader. We can use up to eight textures in our shader. We call glActiveTexture for each of the first three slots and then bind the appropriate handle to the texture. Then in the call to glUniform1i(glGetUniformLocation(programID, “tex”), 0) we also set the second argument to match the slot we picked earlier. That’s all about it. The rest of this function mirrors the other one.

One last thing, lets talk about the camera code. The ColorTrackingCamera class is by Brad Larson and is from his ColorTracking project you can get here. The class sets up a capture session exactly as we did in the Core Image code in other posts on this site.

We set our OpenGLView class as the delegate for the capture session and here’s what we do with the pixel buffer data to get it into our texture object:


- (void)processNewCameraFrame:(CVImageBufferRef)cameraFrame;
{
	CVPixelBufferLockBaseAddress(cameraFrame, 0);
	int bufferHeight = CVPixelBufferGetHeight(cameraFrame);
	int bufferWidth = CVPixelBufferGetWidth(cameraFrame);
    
	// Create a new texture from the camera frame data, display that using the shaders
	glGenTextures(1, &_cameraTexture);
	glBindTexture(GL_TEXTURE_2D, _cameraTexture);
	glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
	glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
	// This is necessary for non-power-of-two textures
	glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
	glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
	
	// Using BGRA extension to pull in video frame data directly
	glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, bufferWidth, bufferHeight, 0, GL_BGRA, GL_UNSIGNED_BYTE, CVPixelBufferGetBaseAddress(cameraFrame));
    
	[self renderNow];
	
	glDeleteTextures(1, &_cameraTexture);
    
	CVPixelBufferUnlockBaseAddress(cameraFrame, 0);
}

We set up a texture the same way we did in the beginning of our program. The glTexImage2D call takes the data from the pixel buffer and puts it into a texture that openGL can use.

We also call the renderNow call from this callback so that we get a render loop each time we get a new camera frame. We don’t need to update the game any more often than that and it appears to give about 30 frames per second on an iPhone 4S or iPad 2. I’m not sure of the performance on a previous gen device, but I suspect it would do fine on an iPhone 4.

That’s about all for this post. I realize that I didn’t explain many of the basics of openGL, so if you are new to the API, here are the places I used to learn most of the things I didn’t cover here. I’d start with Jeff LaMarche’s openGL resources. These are excellent and easy to follow. Then I’d do in no particular order Ray Wenderlich’s OpenGL ES 2.0 for iPhone parts 1 and 2 and Apple’s OpenGL ES Programming Guide for iOS, especially the part about Drawing as it talks about setting up all these frame buffers. There are also some books out there, Philip Rideout’s iPhone 3D Programming, though this book is a C++ book, which I’m useless with. Also, this book, also C++, has some good things in it as well. I think the blog posts are easier to follow and get me to and understanding quicker (than the books), but that could just be me.

I’d like to write a bunch more stuff about openGL. If this is useful let me know in the comments. It will help me learn and I think there’s a lot of cool things to be done with shaders in Cocos2D. Bump mapping, lighting, cool pixellation shader, edge detection, gpu based particle engines, maybe come gpu physics, I don’t really know, but I think it could be cool. If anyone has any suggestions about good openGL resources let me know that, too.

Here’s another link to the finished project..

3 thoughts on “My Quest to Learn OpenGL

  1. Reply a fellow openGL newbie May 16, 2012 2:57 pm

    Any chance you can post the code for this?

  2. Reply a fellow openGL newbie May 16, 2012 2:57 pm

    As in the Xcode project…

    • Reply admin May 16, 2012 8:36 pm

      There’s a link to the project towards the beginning of the post, but I’ve added another link in the last paragraph. Enjoy.

Leave a Reply