Cg Programming/Unity/Cutaways

This tutorial covers discarding fragments and front-face and back-face culling. This tutorial assumes that you are familiar with vertex output parameters as discussed in Section “RGB Cube”.

The main theme of this tutorial is to cut away triangles or fragments even though they are part of a mesh that is being rendered. The main two reasons are: we want to look through a triangle or fragment (as in the case of the roof in the drawing to the left, which is only partly cut away) or we know that a triangle isn't visible anyway; thus, we can save some performance by not processing it. GPUs support these situations in several ways; we will discuss two of them.

Very Easy Cutaways

The following shader is a very easy way of cutting away parts of a mesh: all fragments are cut away that have a positive $y$ coordinate in object coordinates (i.e. in the coordinate system in which it was modeled; see Section “Vertex Transformations” for details about coordinate systems).

Here is the code:

Shader "Cg shader using discard" {
   SubShader {
      Pass {
         Cull Off // turn off triangle culling, alternatives are:
         // Cull Back (or nothing): cull only back faces 
         // Cull Front : cull only front faces
 
         CGPROGRAM 
 
         #pragma vertex vert  
         #pragma fragment frag 
 
         struct vertexInput {
            float4 vertex : POSITION;
         };
         struct vertexOutput {
            float4 pos : SV_POSITION;
            float4 posInObjectCoords : TEXCOORD0;
         };
 
         vertexOutput vert(vertexInput input) 
         {
            vertexOutput output;
 
            output.pos = UnityObjectToClipPos(input.vertex);
            output.posInObjectCoords = input.vertex; 
 
            return output;
         }
 
         float4 frag(vertexOutput input) : COLOR 
         {
            if (input.posInObjectCoords.y > 0.0) 
            {
               discard; // drop the fragment if y coordinate > 0
            }
            return float4(0.0, 1.0, 0.0, 1.0); // green
         }
 
         ENDCG  
      }
   }
}

When you apply this shader to any of the default objects, the shader will cut away half of them. This is a very easy way of producing hemispheres or open cylinders.

Discarding Fragments

Let's first focus on the discard instruction in the fragment shader. This instruction basically just discards the processed fragment. (This was called a fragment “kill” in earlier shading languages; I can understand that the fragments prefer the term “discard”.) Depending on the hardware, this can be a quite expensive technique in the sense that rendering might perform considerably worse as soon as there is one shader that includes a discard instruction (regardless of how many fragments are actually discarded, just the presence of the instruction may result in the deactivation of some important optimizations). Therefore, you should avoid this instruction whenever possible but in particular when you run into performance problems.

One more note: the condition for the fragment discard includes only an object coordinate. The consequence is that you can rotate and move the object in any way and the cutaway part will always rotate and move with the object. You might want to check what cutting in world space looks like: change the vertex and fragment shader such that the world coordinate $y$ is used in the condition for the fragment discard. Tip: see Section “Shading in World Space” for how to transform the vertex into world space.

Better Cutaways

If you are not(!) familiar with scripting in Unity, you might try the following idea to improve the shader: change it such that fragments are discarded if the $y$ coordinate is greater than some threshold variable. Then introduce a shader property to allow the user to control this threshold. Tip: see Section “Shading in World Space” for a discussion of shader properties.

If you are familiar with scripting in Unity, you could try this idea: write a script for an object that takes a reference to another sphere object and assigns (using GetComponent(Renderer).sharedMaterial.SetMatrix()) the inverse model matrix (GetComponent(Renderer).worldToLocalMatrix) of that sphere object to a float4x4 uniform parameter of the shader. In the shader, compute the position of the fragment in world coordinates and apply the inverse model matrix of the other sphere object to the fragment position. Now you have the position of the fragment in the local coordinate system of the other sphere object; here, it is easy to test whether the fragment is inside the sphere or not because in this coordinate system the default Unity spheres are centered around the origin with radius 0.5. Discard the fragment if it is inside the other sphere object. The resulting script and shader can cut away points from the surface of any object with the help of a cutting sphere that can be manipulated interactively in the editor like any other sphere.

Culling of Front or Back Faces

Finally, the shader (more specifically the shader pass) includes the line Cull Off. This line has to come before CGPROGRAM because it is not in Cg. In fact, it is a command of Unity's ShaderLab to turn off any triangle culling. This is necessary because by default back faces are culled away as if the line Cull Back was specified. You can also specify the culling of front faces with Cull Front. The reason why culling of back-facing triangles is active by default, is that the inside of objects is usually invisible; thus, back-face culling can save quite some performance by avoiding to rasterize these triangles as explained next. Of course, we were able to see the inside with our shader because we have discarded some fragments; thus, we should deactivate back-face culling.

How does culling work? Triangles and vertices are processed as usual. However, after the viewport transformation of the vertices to screen coordinates (see Section “Vertex Transformations”) the graphics processor determines whether the vertices of a triangle appear in counter-clockwise order or in clockwise order on the screen. Based on this test, each triangle is considered a front-facing or a back-facing triangle. If it is front-facing and culling is activated for front-facing triangles, it will be discarded, i.e., the processing of it stops and it is not rasterized. Analogously, if it is back-facing and culling is activated for back-facing triangles. Otherwise, the triangle will be processed as usual.

What can we use culling for? One application is to use a different shader for the front faces than for the back faces, i.e. for the outside and the inside of an object. The following shader uses two passes. In the first pass, only front faces are culled and the remaining faces are rendered red (if the fragments are not discarded). The second pass culls only back faces and renders the remaining faces in green.

Shader "Cg shader with two passes using discard" {
   SubShader {

      // first pass (is executed before the second pass)
      Pass {
         Cull Front // cull only front faces
 
         CGPROGRAM 
 
         #pragma vertex vert  
         #pragma fragment frag 
 
         struct vertexInput {
            float4 vertex : POSITION;
         };
         struct vertexOutput {
            float4 pos : SV_POSITION;
            float4 posInObjectCoords : TEXCOORD0;
         };
 
         vertexOutput vert(vertexInput input) 
         {
            vertexOutput output;
 
            output.pos = UnityObjectToClipPos(input.vertex);
            output.posInObjectCoords = input.vertex; 
 
            return output;
         }
 
         float4 frag(vertexOutput input) : COLOR 
         {
            if (input.posInObjectCoords.y > 0.0) 
            {
               discard; // drop the fragment if y coordinate > 0
            }
            return float4(1.0, 0.0, 0.0, 1.0); // red
         }
 
         ENDCG  
      }

      // second pass (is executed after the first pass)
      Pass {
         Cull Back // cull only back faces

         CGPROGRAM 
 
         #pragma vertex vert  
         #pragma fragment frag 
 
         struct vertexInput {
            float4 vertex : POSITION;
         };
         struct vertexOutput {
            float4 pos : SV_POSITION;
            float4 posInObjectCoords : TEXCOORD0;
         };
 
         vertexOutput vert(vertexInput input) 
         {
            vertexOutput output;
 
            output.pos = UnityObjectToClipPos(input.vertex);
            output.posInObjectCoords = input.vertex; 
 
            return output;
         }
 
         float4 frag(vertexOutput input) : COLOR 
         {
            if (input.posInObjectCoords.y > 0.0) 
            {
               discard; // drop the fragment if y coordinate > 0
            }
            return float4(0.0, 1.0, 0.0, 1.0); // green
         }
 
         ENDCG  
      }
   }
}

Remember that only one subshader of a Unity shader is executed (depending on which subshader fits the capabilities of the GPU best) but all passes of that subshader are executed.

On many GPUs, there is a more efficient way to distinguish front and back faces in Cg using a fragment input parameter with the semantic VFACE; see Unity's documentation of shader semantics. However, not all GPUs support this.

Summary

Congratulations, you have worked through another tutorial. (If you have tried one of the assignments: good job! I didn't yet.) We have looked at:

How to discard fragments.
How to specify the culling of front and back faces.
How to use culling and two passes in order to use different shaders for the inside and the outside of a mesh.

Very Easy Cutaways

Discarding Fragments

Better Cutaways

Culling of Front or Back Faces

Summary

Further reading