Petteri's Pontifications
My musings about photography, mostly.
RSS feed
Layers and Masks

Layers and Masks

This lesson is our first foray into what could be called into "advanced photo-processing techniques." Layering and masking can give you extremely fine control over what happens in different areas of the photo, but in order to use them and not get caught, it's important to have an understanding of what's going on.

There are many purposes for using masks, for example, for creating panoramas, manipulating photos (adding or removing stuff), or doing purely artistic stuff. I most often end up using them to manage contrast between different areas of the picture. For example, I might have a sky that's too bright or a ground that's too dark, and I'll want to darken the one and brighten the other, for a more pleasing or more perceptually accurate overall look to the picture. This is what we'll do here. We'll take one picture with this problem, and use layers and masks to address it. However, the selecting, layering and masking techniques are completely general: you can use them for any purpose where you need to combine things from different pictures or make changes to different areas in the picture.

First we'll take the picture, make a mask for the ground I want to lighten, and lighten it. Then we'll use an even fancier technique: we'll take another frame (the same scene but exposed for the foreground, with a slightly shifted focus point, too), and replace the lightened foreground layer with the second frame. Since I shot the frames hand-held, we'll need to nudge it into alignment first. We will use the same mask we created for the first purpose.

What are layers?

Layers are simple to understand. They're just like translucent sheets of plastic with whatever picture is in them, stacked on top of each other. However, unlike ordinary sheets, you can control the degree of opacity (0% is invisible, 100% is completely opaque) and even the blending method -- the mathematical magic that is used to calculate the way the layers are blended with each other.

There are lots of blending methods. I don't even know what all of them do. However, there are a few that are particularly useful: take note for future reference, and try them out. Apart from "Normal," here are some that I like to use:

  1. Darken and Lighten: With Darken, the system will look at each pixel on both layers, and select the darker one. With Lighten, it's the other way around. Lighten works great for lightening shadows and bringing out shadow detail; Darken works for highlights.
  2. Color: This mode has a specific use. When using Smart Blur or Gaussian Blur to tone down color noise (with or without Vincent's Edge Mask technique, see his 123 of Digital Imaging book), set blending mode to Color. This only blur color differences, not lightness differences.
  3. Luminosity: The opposite of Color. Slightly over-sharpen the top layer, set blending mode to Luminosity, and turn down translucency until you get the degree of sharpening you want. Gives a more even, natural appearance than just plain sharpening, although the difference is subtle. Even better, use Vincent's edge mask technique in reverse to sharpen only the detail and leave the noise in the soft areas untouched.
  4. Difference: Need to stack two bracketed frames? Use Difference mode. The misalignments really jump out at you, and you'll see immediately when it "snaps" into alignment.

What's an alpha mask?

What do you do when you want to add something into a picture? You paste it into a second layer and then use the eraser to erase everything except whatever you want to add, so the bottom layer shows through, of course. If you want to leave some translucency, you use a soft-edged eraser or an "airbrush" eraser.

However, if you accidentally erase something you didn't mean to and only notice some time later, you have an annoyance on your hands: you have to either back up and redo everything from where you started, copy-and-paste stuff from the original into the spot you erased and the erase around it again to hide the copy-and-paste, and so on.

An alpha mask works exactly like erasing stuff from the top layer, with one major advantage: you can un-erase stuff just as easily as erasing it.

The idea is very simple. We'll just color-code translucency. White means opaque, black means transparent (the background layers show through completely), and grays mean intermediate degrees of translucency; the darker the more translucent. So, if we take two layers and attach a grayscale image to the second one, the background layer will show through in those places where the attached grayscale image is gray or black, and won't where it is white. This grayscale image is the alpha mask.

Of course, you can apply layer opacity in addition to alpha masking: this just means that the baseline for 100% opacity will shift: set layer opacity to 50%, and the stuff on the white parts of the mask will show up as 50% translucent, and the black ones will still be 100% transparent.

So, if you want to "erase" something from the top layer, just create an alpha mask and make a black area on it in the same position as whatever you want to erase. If you change your mind, just pick some white paint or the eraser and turn the part of the mask white again. For translucency, use varying degrees of gray, for example by using the airbrush tool with either black or white paint.

Technique

Enough theory. Let's look at practice. There are some rather knotty problems with alpha masking. Mostly it's about getting the layers to blend cleanly, with no joins visible. Fortunately, there are a whole number of things you can do to achieve this, progressing from the approximate to the precise. Here's roughly how I usually go about creating a mask:

0. Layer and edit

Create a layer with whatever stuff you want to blend with the background. It could be just a clone of the background layer. Then do whatever edits you need to do to the layer, concentrating only on the area that you are going to leave visible. For example, if you want to pull up shadow detail, play with Levels or Curves until the shadows look right in the area under consideration, and never mind if you blow the higlights in the areas that will be masked out.

We'll use as an example a picture where the sky is exposed just about right, but the foreground is a bit too dark. We'll layer and mask it to selectively lighten the foreground: I duplicated the background layer, then adjusted Curves to pull up the exposure, looking only at the ice in the foreground.

Then we'll do the same again, but this time we'll blend the picture with a lighter exposure of the same scene, instead of doing the deed with curves or levels.

The tonality in the sky got captured nicely, and the silhouette of the islands looks good, but the foreground is a bit dark.

1. Magic wand

Use the magic wand tool to select along the boundary I want for the mask. Use the Tolerance control to your advantage: too low, and it'll select islands of stuff and won't recognize the boundary; too much, and it'll select way over the boundary. I find the "right" setting is usually when it just starts to spill over, selecting some "fjords" in places on the wrong side. Use the magic wand on whichever side of the boundary is more uniform: you can easily invert the selection (or the actual mask) at the end.

Here's what I got after a few tries at clicking near the edge of the ice with the magic wand set to tolerance 16.

2. Expand and Contract

Use Select > Modify > Expand and Contract (by a few pixels) to get rid of any small "islands" left by the magic wand tool, if necessary. (I've actually canned this at 2 pixels into an Action.)

3. Lasso out the fjords

There will very likely be some areas where the magic wand overselected or underselected. Lasso these out with the lasso tool, while holding the shift key down (to select under-selected areas) or the alt/option key down (to de-select overselected fjords): you only need to be precise near the border; once you're clear of it, just draw a big loop.

4. Lasso in and out the big stuff

Use the lasso tool with big loops, pulling it well beyond the edge of the picture, to grab any big areas not selected by the magic wand (or accidentally selected by it). Alternately, use the rectangular marquee tool. Again, shift key down means add to the selection, alt/option, remove from the selection. Remember, you only used the magic wand near the actual tricky edges of the mask: there's probably large chunks of stuff that should be in the mask but aren't. Again, no need to be precise, as you're well within the borders of the selection.

Here's the final selection, after cleanup with expand-contract, the lasso, and the rectangular marquee.

5. Feather

"Blur the edges" of the selection with Select > Feather. About 3-6 pixels is usually good for full-resolution pictures.

6. Create quick mask

Click the quick mask icon to create a quick mask from your selection, if necessary. Make sure you're in the right layer. In Photoshop, the quick mask icon is the one like a rectangle with a circle in dotted line inside it, at the bottom of the Layers palette.

7. Edit the mask if necessary

You may have gotten the opposite of what you wanted, and masked out what you wanted to leave in. If so, simply use Invert to turn the mask into its negative. If there are problem areas, use a small, soft-edged brush to smooth them out -- either with black, to turn them transparent, or white, to turn them opaque. Much of the time, this isn't even necessary.

Contrasty edges can still be tricky. Instead of painstaking pixel-by-pixel work, though, usually a gentle sweep with an airbrush or soft-edged brush will do the trick: here, perfect is the enemy of the good enough, and often close is better than exact. In a worst case, you might have to duplicate the border area into yet another layer, and make a special mask just for it, using either Darken or Lighten modes to get the blend going smoothly.

In Photoshop, you can show the mask by selecting the Channels tab and clicking on the eye box next to the Mask channel. Here's my finshed mask.

8. Adjust opacity

Much of the time, a 100% opaque top layer will look a bit unnatural. Turn down the opacity until the layers blend "naturally." It's a good idea to take a little break here, because your eyes have gotten used to seeing the picture one way, and it'll be tricky to see it with "fresh eyes:" something that looks unnatural at first sight may be just the way it should be after a little while. That's all there's to it!

Here's what I ended up with. I set opacity to about 70%.

Extra credit: Merging bracketed frames

Finally, let's look at a very slightly more involved technique involving layers and masks: merging two bracketed frames. The mask is the same as we just did. The only additional complication is getting the bracketed shots aligned. For this, the Difference blending mode comes in very handy.

1. Paste in bracketed frame

Simple enough, just paste the second frame into a layer on top of the first one.

Here's a second exposure of the same scene. The sky is blown-out near the top, but the foreground looks better.

2. Set blending mode to Difference

The picture will turn into an ugly gray.

After setting blending mode to Difference. Note that the scenes aren't exactly perfectly aligned.

3. Move the top layer into place

Drag the top layer until it aligns with the bottom one: use the arrow tool with the whole layer selected. Once close, nudge with the arrow keys. You'll see the "embossed" appearance of the picture go flat once the layers are in alignment.

Now, the layers are almost aligned. The center buoy looks good, but the top and bottom ones have some "emboss" sticking out... on opposite sides. Needs rotation.

4. Rotate and stretch to fit

If you were a diligent photographer and shot the frames off a tripod using a remote release, this will probably be good enough. However, if, like me, you like to minimize your burden, you'll have hand-held bracketed shots that align almost, but not quite. The most common issue is that the camera was not perfectly level on one or both of the shots: there's an almost imperceptible tilt that throws the frames a pixel or two out of line in some areas. There may also be minute differences in size, due to slightly changed position or even focal point.

So, to adjust these, do Edit > Transform. Then rotate, enlarge, or shrink until the frames snap into alignment.

I had to rotate the top layer this much and nudge it a bit before it snapped into alignment. No "embossed look" anymore, except in the treeline where it doesn't count, as it'll be masked out.

5. Crop

If the frames didn't align perfectly, there will be some slop at the edges. Use the crop tool to crop these out.

6. Mask and blend as usual

After this point, everything goes as before: set blending mode to Normal, select, mask, and blend just like for the single picture.

Here's the finished version done by bracketing, masking, and blending.

Assignments

  1. Select a picture where you'd like to tweak the levels or curves in one area but leave the rest of the picture as it is. Use the techniques described in the lesson to edit, mask, and blend the areas. Present and discuss.
  2. Combine two bracketed frames to extend the dynamic range of the camera as discussed in the Extra Credit section. Present and discuss.
  3. Find a completely different use for layering and alpha masking. Describe, present, and discuss.