Direct3D 11 adds three new pipeline stages, with the goal of enabling efficient tessellation of higher order surfaces. This is the Direct3D 10 pipeline, as shown in “Real-Time Rendering, 3rd Edition”:

The color of each stage indicates whether it is fully programmable (green), configurable (yellow) or fixed function (blue). The stages are described more fully in the “Graphics Processing Unit” chapter of the book. Note that the “Geometry Shader” stage is new to Direct3D 10, but the other stages have been in the pipeline for quite a while.

The Direct3D 11 pipeline adds three new stages between the vertex and geometry shader stages (framed in red). Two of the new stages are programmable (the hull and domain shader stages) and one is configurable (the tessellator stage):

This pipeline operates on meshes represented as a series of surface patches. Triangle and quad surface patches are primitives in Direct3D 11 (there is also a tessellated line primitive). The shape of each patch is defined by a number of control points. These control points are transformed, skinned and / or morphed one by one in the vertex shader.

The hull shader is called for each patch, using the patch control points from the vertex shader as inputs. The hull shader has two main responsibilities. The first is to (optionally) convert the control points from one representation (basis) to another. for example, it can implement the technique introduced in Loop and Schaefer‘s paper “Approximating Catmull-Clark Subdivision Surfaces with Bicubic Patches“. The control points are sent directly to the domain shader, bypassing the tessellator. The hull shader’s second responsibility is to compute appropriate tessellation factors, which are passed to the tessellation stage. This allows for adaptive tessellation, which can be used for continuous view-dependent LOD (level of detail). The tessellation factors are specified per patch edge, and range from 2 to 64. This means that each edge of the patch may be split into at least 2 (and as many as 64) triangle (or quad) edges.

The tessellator is a fixed-function (but highly configurable) stage, which uses the tessellation factors to tessellate (subdivide) the patch into multiple triangle or quad primitives. The tessellator does not have access to the control points – all tessellation decisions are made based on configuration and the tessellation factors passed on from the hull shader. Each vertex resulting from the tessellation is output to the domain shader. Only the patch parametrization coordinates are passed on for each vertex.

The domain shader operates on the patch parametrization coordinates of each vertex separately, although it can also access the transformed control points for the entire patch. The domain shader sends the complete data for the vertex (position, texture coordinates, etc.) to the geometry shader (or the clipping stage if no geometry shader is present). Effectively, it evaluates the surface representation at each vertex. Techniques such as displacement mapping can also be applied by this shader stage.

Although Microsoft gave an example using Catmull-Clark subdivision surfaces, the programmability of the pipeline enables other surface representations to be used. Alternatively, the tessellation stages can be turned off and traditional triangle or quad meshes can be used.