Thursday, April 30, 2015

Postprocessing in OpenGL

One of the key areas I have immense interest is, learning how to do Postprocessing & then eventually using that setup to create some of the techniques we see most of the times in Games like Bloom, Motion blur etc. This was the reason why I decided to finally sit down & try to implement postprocessing using OpenGL. This is something that I achieved using DirectX few years back, but in my quest to learn OpenGL, I am first trying to do those things which I have already done using OpenGL.

As per the conceptual part goes, post processing is nothing but, using an off screen buffer to render out the content then doing some mathematical operations on it & then presenting the modified output as a final output onto the screen. So basically, we add an extra step in between before presenting our content onto the screen. This extra step is what we will be focusing mainly on.

If you want to go one step deeper then technically we are going to render our scene to an off screen buffer whose resolution is same as that of actual framebuffer. Next, we are going to use this off screen buffer as a texture which will be rendered on a screen aligned quad. The main reason for doing all these steps is, we get to do some cool post processing effects as we will soon find out.

For this, again, I started by writing a class for Framebuffer entity with member variables for VBO, VAO, shader attributes & most importantly screen aligned quad vertices.  We also have a FBO (frame buffer object) identifier which will be used later to use the buffer. 
Quad vertices are created in the usual way. Each vertex has position & texture coordinate data. As these are screen aligned vertices, we do not use z coordinate & simply use 0 for it. Also, note that position values are directly specified in NDC space ranging from [-1, 1].

FramebufferSetup() function does all the required setup work. First we create an empty texture object size of which is same as that of backbuffer.

glGenTextures(1, &tbo);
glBindTexture(GL_TEXTURE_2D, tbo);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, 1280, 800, 0, GL_RGB, GL_UNSIGNED_BYTE, NULL);
glBindTexture(GL_TEXTURE_2D, 0);

There are couple of things to note, first, texture data is passed as NULL, reason being we just create the framebuffer object, data will be written into it later when we render to the framebuffer. Second, we have not specified any wrapping method as it is usually not required.
Next part is to create frame buffer object & attach above created texture to it. Before we do that, there are couple of things to know about attachment. There are four types of attachments we can do to framebuffer. They are mainly Color, Depth, Stencil & DepthStencil. Please refer to OpenGL specs to know more in detail about what each option provides. Apart from Framebuffer object, OpenGL also provides what is known as Renderbuffer objects. Renderbuffer objects store data in a native format that opengl understands hence they are extremely fast as compared to framebuffers. Mostly, whenever you want to do texture lookup to fetch the color, we use Framebuffer object. Renderbuffer objects are write only buffers & being very fast they are useful in operations such as buffer switching.
Code below shows how we create both Framebuffer & Renderbuffer objects in the code : 

glGenFramebuffers(1, &fbo);
glBindFramebuffer(GL_FRAMEBUFFER, fbo);

// attach texture object to already bind framebuffer as color attachment
// create render buffer object
glGenRenderbuffers(1, &rbo);
glBindRenderbuffer(GL_RENDERBUFFER, rbo);
glRenderbufferStorage(GL_RENDERBUFFER, GL_DEPTH24_STENCIL8, 1280, 800);
glBindRenderbuffer(GL_RENDERBUFFER, 0);

// attach render buffer object as depth & stencil attachment to framebuffer

// Finally, check if framebuffer is complete
std::cout << "Frame Buffer status error!" << std::endl;

glBindFramebuffer(GL_FRAMEBUFFER, 0);

As you can see, texture object created earlier was assigned as a COLOR attachment & separate render buffer objects was created for Depth & Stencil attachment.
Once that is done, we create VAO for screen aligned quad as follows: 

glGenVertexArrays(1, &vao);

glGenBuffers(1, &vbo);
glBindBuffer(GL_ARRAY_BUFFER, vbo);
glBufferData(GL_ARRAY_BUFFER, 6*sizeof(VertexPT), quadVertices, GL_STATIC_DRAW);

posAttrib = glGetAttribLocation(m_pShader->GetShaderID(), "in_Position");
glVertexAttribPointer(posAttrib, 3, GL_FLOAT, GL_FALSE, sizeof(VertexPT), (void*)0);

texAttrib = glGetAttribLocation(m_pShader->GetShaderID(), "in_TexCoord");
glVertexAttribPointer(texAttrib, 2, GL_FLOAT, GL_FALSE, sizeof(VertexPT), (void*)offsetof(VertexPT, uv));


I created helper functions, BeginRenderToFramebuffer() & EndRenderToFramebuffer() so that binding & unbinding framebuffer becomes easy. Here are those functions:

void Framebuffer::BeginRenderToFramebuffer()
       glBindFramebuffer(GL_FRAMEBUFFER, fbo);

       glClearColor(0.3f, 0.3f, 0.3f, 1.0f);


void Framebuffer::EndRenderToFramebuffer()
       glBindFramebuffer(GL_FRAMEBUFFER, 0);

Most important function to check is RenderFramebuffer(), so lets have a look at that: 

void Framebuffer::RenderFramebuffer()

       // No need of any depth testing while rendering a single scene aligned quad...

       //glPolygonMode(GL_FRONT_AND_BACK, GL_LINE);


       glBindTexture(GL_TEXTURE_2D, tbo);

       glDrawArrays(GL_TRIANGLES, 0, 6);


Now that all the basic starting setup is done, let us look at the steps involved for Render to texture operation.
a.      Bind newly created framebuffer & render scene as usual.
b.      Bind back the default framebuffer.
c.       Draw the screen aligned quad with new framebuffer set as active texture to it.
I have a scene class which manages the main rendering. Here is how I am managing above states:




This is just to give you an idea how helper function was used to turn on & off rendering to off screen buffer. 

So far, we have taken care of rendering actual content onto the offscreen framebuffer which is used as a texture to render a screen aligned quad. Why do we do all this? Well, the trick is, now we actually have our scene as a texture! So any fancy operations we do on the texture, it appears to be done on an entire scene, which definitely adds to the value. We have used basic shader so far to render a screen aligned quad with framebuffer texture. Here is the basic version of the shader: 

Vertex shader simply outputs the position as it is, remember vertices for the quad are already specified in NDC.

void main()
       gl_Position = vec4(in_Position, 1.0f);
       vs_outTexCoord = in_TexCoord;

And here is the pixel shader which reads that & blits the output.

void main()
       outColor = vec4(texture(screenTexture, vs_outTexCoord));

All is good so far. Now, we can apply some operations on the final color before they are applied to the outColor variable as a final fragment color. This operations can be anything, however, to make sense out of it, we will do simple monochrome effect by premultiplying pixel values by known constants. 

First let us have a look at scene rendered on a quad without any post processing applied over it. 

We apply following function over a scene texture to change final output. 

vec4 BlackWhiteFilter()
       vec4 screenColor = vec4(texture(screenTexture, vs_outTexCoord));
       return vec4((0.2126f * screenColor.r + 0.7152f * screenColor.g + 0.0722f *     screenColor.b) );

And here is the same scene with Black & White post processing applied over it.

Now let us take a look at kernel based post processing effects. These effects act on each pixel considering effect of surrounding pixels color values as well. How much influence each surrounding pixel color has on the current pixel color is decided by the weighted matrix which acts on each pixel. There are tons of resources explaining kernel based post processing, but I have tried to keep the definition simple. In practice, what we do is add small offset to a texture coordinate in surrounding directions of the current pixel & combine the result based on matrix. This convolution matrix can be 3x3 or 5x5 or of any size, it’s the performance vs output consideration that we have to do while using this technique. Here is how we do this in glsl shader code: 

const float offset = 1.0f/3000.0f;
const vec2 offsets[9] = vec2[] (
                                  vec2(-offset, offset),
                                  vec2(0, offset),
                                  vec2(offset, offset),
                                  vec2(-offset, 0),
                                  vec2(offset, 0),
                                  vec2(-offset, -offset),
                                  vec2(0, -offset),
                                  vec2(offset, -offset)

Here offset specifies how much away we are sampling from current pixel. Consider Edge detection kernel code :
vec4 EdgeDetectionFilter(vec3 inSample[9])
       float kernel[9] = float[] (
                                   1, 1, 1,
                                    1, -8, 1,
                                   1, 1, 1

       vec3 color;
       for (int i = 0; i < 9; i++)
              color += inSample[i] * (kernel[i]);

       return vec4(color,1);
Where we call above function in fragment shader’s main function as follows,

// Sample current pixel with the offset from current location
vec3 pixSample[9];
       for (int i = 0; i < 9; i++)
              pixSample[i] = vec3(texture(screenTexture, + offsets[i]));

       outColor = EdgeDetectionFilter(pixSample);

As you can see we send current pixel sample with an offset to function with kernel operation, it is then convoluted and result is sent back. Here is the output of Edge detection filter:

Here is the output with Sharpen filter:

One of the other interesting way of using post processing shader is as follows. Instead of rendering full color to framebuffer, we will simply render z depth values & then we will apply Edge detection filter on it to receive pure edge of the object. We turn off skybox rendering just to showcase the edge clearly.

We can change the thickness of the edge by simply reducing the offset by which we apply our kernel. Here is the output with offset set to 1/300. 

All in all, possibilities are immense with postprocessing. Some of the effects such as Bloom, motion blur use post processing extensively. Next time I will try and see if Bloom can be quickly implemented. I am also planning to introduce Material framework in the code which will make it easy to write some of the new physically plausible shading paradigm. More on this in coming posts. Stay tuned.