5 Pro Photoshop Masking Tricks You Don’t Know (Probably)
416K views 22.4K likes 342 comments
Jump to a step in the video (7)
Click a time to play the video from there.
Your choices stick across every tutorial on this device.
Photoshop masking tricks like a hand-painted fur brush and a Channel-Based Mask solve masking problems no single selection tool can fix.
As a creative professional with 22 years of experience, especially now with my work creating posters in the Hollywood industry, I’ve picked up a few pro-Photoshop masking tricks. They include advanced group masking techniques I’m sure you haven’t seen anywhere else!
Let’s jump right into it!
The techniques below all work in Photoshop 2023 (version 24.0 or later) or the current version. The video was recorded in December 2022, right after Photoshop 2023’s general release.
Key Takeaways
- The Minimum filter (Filter > Other > Minimum) removes the leftover white outline a Photoshop mask often leaves along an edge.
- A Vector Mask built from a work path gives an object clean, razor-sharp edges that a pixel mask can’t match.
- Painting fur or hair back into a mask with a custom brush beats trying to select every strand.
- A Channel-Based Mask avoids the transparency and missing layer-style problems a blending mode like Screen creates.
- The Knockout option in the Layer Style dialog can punch a hole through a group while keeping a text layer fully editable.
- Building a knockout mask from separate layers, instead of painting directly on one mask, keeps every element in the composite adjustable later.
Table of contents
- What You Will Learn
- Requirements
- Photoshop Masking Tricks Keyboard Shortcuts
- Trick 1: Minimum Filter
- Trick 2: Vector Masks from Selection
- Trick 3: Custom Brushes for Masking
- Painting the Fur Back into the Photoshop Mask
- Trick 4: Channel-Based Masking
- Building the Channel-Based Masking Trick
- Trick 5: Knockout Group Masking
- Fixing the Background Layer for a Deep Knockout
- Building a Multi-Layer Knockout Mask Group
- Adding Blend If and Splatter Masks
- Frequently Asked Questions
- Conclusion
- Related Links
What You Will Learn
- How to remove the white outline a layer mask leaves with the Minimum filter
- How to turn a selection into a clean Vector Mask with the Object Selection Tool
- How to build and paint with a custom fur brush for hair and fur edges
- How to create a Channel-Based Mask for a subject with a complex edge
- How to use the Knockout option to mask a group while keeping a text layer editable
Requirements
Before you start
- Photoshop 2023 (version 24.0 or later) or the current version
Photoshop Masking Tricks Keyboard Shortcuts
| Action | Windows | macOS |
|---|---|---|
| Make a work path from a selection (Paths panel) | Alt | Option |
| Turn a path into a Vector Mask (Layers panel) | Ctrl | Command |
| Load a channel as a selection | Alt Ctrl 6 | Option Command 6 |
| Load a layer as a selection (layer thumbnail) | Ctrl | Command |
| Fill with background color (black) | Ctrl Backspace | Command Delete |
| Move a layer up in the stack | Ctrl ] | Command ] |
| Move a layer down or out of a group | Ctrl [ | Command [ |
| Split Blend If sliders for a smoother transition | Alt | Option |
| Merge visible layers | Ctrl E | Command E |
| Brush Tool | B | B |
| Adjust brush size | [ / ] | [ / ] |
| Rotate the brush | Left Arrow / Right Arrow | Left Arrow / Right Arrow |
| Clip a layer to the one below | Ctrl Alt G | Command Option G |
| Temporarily switch to the Eyedropper while painting | Alt | Option |
Trick 1: Minimum Filter
When you create a mask, you might see a white outline around the edges.
To remove this outline, go to Filter > Other > Minimum.
Select Roundness from the dropdown and drag the Radius slider to the right to remove the white outline.
Trick 2: Vector Masks from Selection
Vector masks are perfect for objects with smooth edges, but they can be difficult and time-consuming to create. Here’s a trick to speed up the process.
Activate the Object Selection Tool. From the Options bar, enable the Object Finder.
The subjects in the image will now highlight as you hover over them. Click on one to make a selection.
Go into the Paths panel, hold Alt (Windows) or Option (macOS), and click this icon.
In the box that appears, input a number between 0.5 and 10.
When you create a vector mask, use the fewest anchor points possible while maintaining the object’s shape.
Lower values give you more anchor points, while higher values give you fewer.
In this example, a value of 4 works great.
From the Layers panel, hold Ctrl (Windows) or Command (macOS) and click on the Layer Mask icon to create a Vector Mask.
Now, you can use the Curvature Pen Tool and all other vector tools to edit the path.
When you’re done, you’ll have an object with smooth, razor-sharp edges.
Trick 3: Custom Brushes for Masking
When you apply a mask to anything with hair or fur, the edges may not look great. Especially when it’s on a complex background.
One solution to this problem is to paint the mask instead of using traditional masking techniques. This is one of the best-kept secrets in the Hollywood industry.
To do this, you’ll need a custom fur brush. Let’s create one from scratch!
Use the Curvature Pen Tool on a white background with black as your foreground color.
From the Options bar, set the mode to Shape.
Then, click to add a vector point on top and bottom of the canvas, and then close the path by clicking on the first point.
Next, click on the center of the line to add an anchor point and drag out and
do the same on the other side.
When you have something that looks like a strand of hair, you can convert it into a Photoshop Brush.
Go to Edit and choose Define Brush Preset.
Give the brush a name and Press OK.
Then, go into the Brush Settings by clicking on the icon indicated in the image below.
You may use my settings as a starting point, but you can adjust the Spacing, Shape Dynamics, Scattering, and Count to fit your needs.
First, increase the Spacing. Notice the preview of the brush here.
Keep an eye on it as you change the brush settings.
Then, enable the Shape Dynamics.
Increase the Size Jitter to about 12% to make each strand slightly different lengths.
Add a tiny bit of Angle Jitter, and set it to 1% to slightly rotate each strand.
Set the Control to the Direction, so the fur follows the direction you paint along.
Next, enable the Scattering. Increase the Scatter to 20% and Count to 6 to add more strands.
When you’re happy with your brush, you can save it.
Click on the flyout menu and choose New Brush Preset and give it any name you want.
Painting the Fur Back into the Photoshop Mask
Want more brushes to build from? Download over 1,000 free Photoshop brushes and add them to your library.
Now, go back into your working document, and let’s get the mask ready.
First, we need to contract the mask, so you have enough room to paint the fur back in.
Go into Filter > Other > Minimum.
Eight pixels work great in this case and Press OK.
Make sure that white is your foreground color. Zoom in to see the edge and paint.
Remember, you can use the left and right Brackets to adjust your brush size and use the left and right arrow keys to rotate the brush.
Keep painting along the edge until you’re happy with the results.
This looks much better than any selection you can make.
However, you need to match the blur on your mask as well.
Go into Filter > Blur > Gaussian Blur.
Match the blur of the image as best as you can and press OK.
Now the edge of the mask looks fantastic.
Trick 4: Channel-Based Masking
How do you mask this pile of snow to make it seem like it’s sitting on these wooden panels?
You might use a Blending Mode, such as Screen.
If you select Screen, black will disappear, and you’ll keep only the bright pixels.
See the full guide to Photoshop blending modes for how Screen and the other modes calculate their results.
But, we run into two problems.
One, the pile of snow becomes somewhat transparent, and we can see the background, which is not very good.
This is just the blend, which means we can’t apply a layer style to the edges of the snow, only to the edges of the layer.
But one trick that can come in handy, especially when dealing with complex images, is using a Channel-Based Mask.
Building the Channel-Based Masking Trick
A Channel-Based Mask allows you to take advantage of the unique color contrasts in different channels of your image, giving you more control over the masking process.
Here’s how to create one.
Go into the Channels panel and select the channel that has the most contrast between the foreground and background.
In this case, they’re all the same, so you can select any channel you want.
Duplicate the channel by dragging it into the New Channel icon.
Go into Image > Adjustment > Levels.
Adjust the levels by dragging the slider to make the background black and the foreground white.
Create a selection out of the active pixels by holding Alt (Windows) or Option (macOS) and clicking on the right side of the channel you made.
In this case, the shortcut is AltCtrl6 (Windows) or OptionCommand6 (macOS), and it creates a selection out of the white pixels in that channel.
Go into the RGB channel and back into the Layers panel and click on the Layer Mask icon to create a mask.
With this Channel-Based Mask, you can make your snow appear like it’s sitting on the wooden panels without any transparency or blend issues.
You can even double-click on the side of the layer and add a Drop Shadow to create the illusion that the snow is actually sitting on the panels.
For more control over how a shadow sits on a surface, see this guide to building customizable drop shadows in Photoshop.
By using this technique, you can get more control over your masking process and achieve better results.
Trick 5: Knockout Group Masking
Knockout Group Masking is a powerful technique in Photoshop that allows you to create a mask based on a text layer while keeping the text editable.
This is done by using the Knockout feature in the Layer Style window.
First, take a look at the Layers panel. We have a background image of New York, a color lookup Adjustment Layer, and a group called Design.
Inside the Design group, we have a sticker that has been masked out already and has a layer style, creating an outer shadow.
We also have a text layer with the words “I love New York.”
The goal is to create a mask based on the text layer to see the background
through the sticker layer.
One way to do it is to create a mask with Black text. As seen in the screenshot below.
The downside is that this method does not give us an editable text.
For example, if you want to change the text to “I love San Francisco,” the mask must be recreated from scratch.
To keep the text editable and create a mask simultaneously, double-click on the side of the text layer to bring up the Layer Style window.
From the Knockout drop-down, select the Shallow option, then bring the Fill opacity to zero.
This will give you the same result as creating a layer mask on the sticker layer, but the text remains editable.
If you change the Knockout option from Shallow to Deep, the hole is punched all the way down to the background layer. It even makes a hole through the Color Look-up Adjustment Layer.
If you try this and see transparent pixels, then you don’t have a background layer.
Fixing the Background Layer for a Deep Knockout
To make sure the bottom-most layer is a Background layer,
go into Layer > New > Background from Layer.
Open the Layer Style window by double-clicking on the side of the text layer.
Change the option from Deep to Shallow.
This will create a hole that only affects the contents of the group.
To move layers up in the layer stack, you can use the keyboard shortcut Ctrl] (Windows) or Command] (macOS).
This will cut a hole in the Color Lookup Adjustment Layer, revealing the original color.
To move the layer out of the group, press Ctrl[ (Windows) or Command[ (macOS).
Building a Multi-Layer Knockout Mask Group
Another way to use the knockout method is to create a new group called “Knockout Mask” within the Design group. Double-click on the side of the group to bring up the Layer Style window.
Change the Knockout to Shallow, so this group only affects its parent group.
Change the fill Opacity to zero and press OK.
Anything placed inside this group will affect the text and the sticker layer.
Here’s how it’s beneficial.
Create a new layer and enable the brush tool by pressing B (Windows) or B (macOS).
Select a brush to create a torn-edge effect.
Paint in black or white, as the color doesn’t matter.
This method is more beneficial as the pixels on this layer act as a mask, known as knockout pixels. This allows you to create the knockout effect using a brush.
Adding Blend If and Splatter Masks
To create even more interesting masks, bring in a layer and use Blend-If.
Drag an image with cracks into the design folder and make it larger to cover the entire sticker.
Then, drag it into the Knockout mask group. It will disappear because it is now covered by all of the pixels in this layer.
Double-click on the side of the layer to bring up the Blend-If options.
This allows you to hide and show pixels based on their brightness.
Hide the white wall by clicking and dragging the slider to the left.
Hold Alt (Windows) or Option (macOS) to split the points and create a smoother transition.
For the full walkthrough of every Blend If slider and setting, see this complete guide to Blend If in Photoshop.
Now you have the cracks as part of your mask.
You can also use splatter brushes to create more holes by selecting them and creating a new layer.
This method is beneficial because all three layers creating this mask are separate.
You have control over each layer, and you can easily make adjustments.
If you take it to your client, boss, or art director, and they’re unhappy with the results, you can easily disable the splatters and keep the cracks. You can bring everything back or hide it. Everything is editable!
By using these five tricks, you’ll be able to create professional-looking masks in Photoshop with ease. Whether working with hair, fur, or smooth or torn edges, these techniques will help you achieve the perfect mask every time.
Frequently Asked Questions
The Minimum filter, found under Filter > Other > Minimum, contracts a mask so it no longer shows a thin white outline around the edges. Set it to Roundness and raise the Radius until the outline disappears.
Make a selection with the Object Selection Tool and its Object Finder, then hold Alt (Windows) or Option (macOS) and click the make-work-path icon in the Paths panel. Hold Ctrl (Windows) or Command (macOS) and click the layer mask icon to turn that path into a Vector Mask.
Selecting individual strands of hair or fur rarely looks convincing. Painting the mask instead, with a custom brush built from a single curved stroke, keeps the edge soft and natural.
Draw one hair strand with the Curvature Pen Tool set to Shape mode, then choose Edit > Define Brush Preset. Open Brush Settings and add Spacing, Shape Dynamics, and Scattering until the strokes look like scattered strands.
A Channel-Based Mask uses the channel with the most contrast between the subject and the background, boosted with Levels, to build a selection. It avoids the transparency and missing-layer-style problems a blending mode like Screen creates.
Duplicate the channel, then adjust it with Levels until the subject is white and the background is black. Press AltCtrl6 (Windows) or OptionCommand6 (macOS) to load it as a selection.
Knockout, set to Shallow or Deep with the Fill opacity at zero, punches a hole through the layers below a shape or text layer. Unlike painting a mask by hand, the text or shape stays fully editable.
A Knockout Mask group lets you build a mask from several separate layers, brush strokes, cracks, and splatters, and keep each one editable. You can hide or bring back any single element without redoing the whole mask.
Conclusion
These five Photoshop masking tricks solve problems a single selection or one pixel mask can’t. The Minimum filter cleans up outlines, a Vector Mask gives razor-sharp edges, and a painted fur brush handles hair. A Channel-Based Mask isolates a complex subject, and a knockout group keeps every element in a composite fully editable.
Start with whichever trick matches the edge you’re fighting right now, then watch the full video above for the exact settings used on each example.
Comment on the video if you have a question about any of these techniques.
Selections and Masking Mastery
Complete course includes 2.5 hours of step-by-step lessons, 6 structured modules, downloadable practice files, and custom Photoshop brushes. Everything you need to master selections and masks in one place.
Keep going
