Short answer: A set extension is visual effects work that adds the rest of a place to footage of a set that was only partly built. The real set stays in the frame, and the missing walls, streets or skyline are painted or built digitally and matched to the camera. It can be a strip of sky or most of the frame.
The term describes the result, not the method. A painted strip of sky and a modelled city with a moving camera inside it are both set extensions, and the price follows the method. Whether to extend a set at all, or build it, is a separate question, covered in when a set extension is cheaper than building the set.
What does set extension mean in VFX?
In visual effects, a set extension is any digital addition that continues a physical set past the edge of what was built. The actors perform on real floors, in front of real walls, under real light. Past a line somewhere in the frame, the place carries on as something made in post.
The idea is older than computers. Wikipedia's history of matte painting dates the first known matte painting shot to 1907, when Norman Dawn painted crumbling California missions on glass to complete the real scenery in front of his camera. The glass is gone, as what a matte painting is now explains. The job of finishing a place the production could not afford, or could not reach, is the same.
What separates a set extension from other environment work is that it has to touch something real. A fully digital shot answers only to itself. A set extension sits next to a built wall in the same frame, under the same light. And the join between them is where the audience looks without knowing it.
Partial and full set extensions
People use two rough categories, and the line between them is loose.
A partial set extension adds a part of the frame. The built set carries the shot, and the extension finishes it: the upper floors of a street, the far end of a corridor. On Warfare, according to befores & afters, the plan was to build just the one house and digitally do the rest. The build grew to eight houses in an airfield car park, surrounded by two bluescreens that the visual effects supervisor described as 20 feet high by 120 feet wide. Cinesite then extended the street and replaced some skies.
A full set extension is the same idea taken almost to the edge of the frame. Only the foreground the actors touch is real, and everything behind them is digital. For a wide reveal in the SNL trailer known as HBO Mario Kart, the team shot the foreground on a practical set with a blue screen in place to allow for what the artist called a massive set extension. The CG world behind the practical set was most of the picture.
Most shows mix the two inside a single sequence. For any one shot, ask how much of the frame is new, and whether the camera moves through it.
Three ways to make one: 2D, 2.5D and full 3D
The kind of extension is about how much frame it covers. The method is about how it is made, and there are three common ones.
A 2D painted extension is a flat image laid into a shot where the camera does not move or barely moves. It is usually painted over photographic material. It is the cheapest kind and the quickest. It is also the one that breaks first, because a flat image cannot show depth once the camera starts to travel.
A 2.5D projection takes that painting, splits it by distance, and projects the pieces onto simple geometry placed where those surfaces would really be. When the camera moves, near things pass in front of far things the way they would in a real place. Most modern matte painting works this way. The reasons a flat painting fails on a moving camera, and what depth costs, are in why a painted environment falls apart when the camera moves.
A full 3D extension models the missing part of the place, textures it, lights it and renders it. It costs the most to start. It's the only option when the camera travels far, the light has to change, or something inside the extension has to move. The trade-off is set out in matte painting versus CG environments.
These methods are not exclusive. On the same HBO Mario Kart shot, the road leading into the CG world carried a digital matte painting made from imagery of the practical set, projected over 3D geometry. A painting on top of a build is a common way to close the gap between what was shot and what was rendered.
How a set extension is shot
The extension is made in post, but most of what decides its quality happens on the day.
A blue or green screen behind the build. Wherever the set stops and the extension begins, a coloured screen makes the edge easy to separate. Wikipedia's article on chroma key notes that green is favoured in digital work because bright green is less likely to appear in the foreground, and that blue is used when the subject is mostly green, such as plants. It also describes spill, the screen colour bouncing onto the actors, which a larger screen placed farther away helps avoid. Costume matters too. Hybride's supervisor told befores & afters that on Obi-Wan Kenobi, cloth costumes were a lot more forgiving against bluescreen than the reflective armour on The Mandalorian, which throws blue reflections everywhere.
Tracking markers. A moving camera has to be reconstructed in software before anything can be added to the shot. The same chroma key article describes markers placed on the screen for this, such as a painted grid or tennis balls attached to the wall. The process is called match moving, which Wikipedia describes as a way to insert new elements into live action footage so they sit in the right place and move the way the photographed scene moves.
A clean plate. This is a pass of the same framing with the actors and rigs out of the shot. It gives the artist the real background behind anything that has to be removed or rebuilt along the join. What it is and why it never quite matches is covered in what a clean plate is for.
Records of the camera and the light. Lens, height, focus distance and where the sun or the key light sat. Each is easy to write down on the day and hard to recover from the frames.
How is a set extension done in Nuke?
Nuke, Foundry's compositing program, is a common place to assemble one. The details vary by studio, but the core steps usually run in this order.
First, the camera is tracked. Foundry's documentation describes its CameraTracker node as a tool that lets you create a virtual camera whose movement matches that of your original camera. Moving things in the plate, such as actors, are masked out so they do not confuse the solve.
Second, simple geometry is placed in the tracked scene where the missing walls, buildings or terrain would stand.
Third, the painting is projected onto that geometry. The Project3D node, in Foundry's words, projects an input image through a camera onto the 3D object. Render the projected geometry through the shot camera, and the painting moves with the plate.
Last, the extension is keyed and composited against the screen, graded to the plate, and given the plate's grain and lens softness so the join stops reading as a join. Full 3D extensions are often rendered in a separate program first and then brought into Nuke for this final step, which is how one of the HBO Mario Kart artists describes his own workflow.
What usually goes wrong
Copying the set instead of continuing it. The built part sets a style, and the obvious move is to repeat it. Hybride's supervisor said of extending rock formations that if you take the set design and just extrapolate it, everything is going to look flat or repetitive. He said the first pass felt like tiling of what was on set. Real places vary, and an extension has to vary too.
Ignoring what the set already gives you. The better habit is to borrow from the set itself. On Warfare, the supervisor said the textures sourced from the built houses were what we use to duplicate and replicate out the rest of the build in CG. Materials taken from the real set, under the shoot's own light, start much closer to the plate than anything sourced elsewhere.
A camera move nobody planned for. A shot bid as a flat painting that later gets a move in the edit has changed method, and usually price.
A screen in the wrong place. If the build ends but the screen does not cover the gap, the extension has to be rotoscoped, traced by hand frame by frame, before it can be laid in.
If you have a shot to extend
Chumak Studio, in Toronto, paints environments for film and television, including set extensions. Anastasiia Chumak, Art Director at Chumak Studio, has credits since 2016, among them Black Phone 2 as "Senior DMP artist, Folks VFX", where the Folks VFX environment work blended photographed Colorado landscapes with set extensions. The services are described on the digital matte painting page. If you have a plate and are not sure which kind of extension it needs, send the frame and the camera notes and you will get a straight answer about the method, including when a different one would serve the shot better.
