I'm just going to use some of my old icons as examples, but let's start out with the basics. (You can see the names of the different palettes here.)
The easiest way to add a layer mask is to click on the layer you want to add the mask to and click the "Add layer mask" button in the layer palette.
As you can see, a mask has already been added in this case.
You can very easily tell the difference between when the layer is selected and when the layer mask is selected. First, the one that's selected has a white frame. Second, in mask mode the colors in the tool palette switch to black and white. I moved the tools palette next to the layer palette so it would be easier to tell:
You change between layer mask and layer simply by clicking on the them in the layer palette.
Masks are basically a different way of erasing, without using the eraser tool. When you want to erase something you simply add a layer mask to the layer, make sure the mask is selected, pick a brush, select the color black and start painting on the canvas wherever you want to erase something.
And the layers in question:
The good thing is that if you realize later on that you've made a mistake/don't like it anymore/want to try something different you can just go back and paint the areas that you want to undo white. Like this icon? I made one version where the dividing line between the two layers was too straight and I wanted a feel of the book world melding with reality, so I went back after a few days and just repainted the layer mask.
You don't have to use black and white, any shade of gray works. And if you try to pick an actual color when a mask is indicated it'll just become gray because colors don't work on masks. Gray-scale works sort of like lowered opacity on the eraser tool, with one major difference: with the eraser tool, if you paint in the same spot again with lowered opacity it'll add to it (so if the opacity is 50% and you double click the "eraser strength" will become 75%). That is not the case with shades of gray on a mask layer.
Knowing about masks is probably most useful with layers that already have layer masks, such as color fill and gradient layers. Here's the difference mask use made in one icon:
Another good thing about masks: if you want the same mask on two layers you can just duplicate the layer and then change either color, gradient or blending mode.
Here's one where I used both masks and grouping to create the border.
I used these stock images in the borders:
Layer 8 and all its copies are the second image, layer 6 (blending mode: Difference) is the first one and the sky (blending mode: Darken), as you can see, is grouped with another regular border brush layer.
Which brings me to a final note: grouping. Doesn't really have a place in a masks tutorial, but I really enjoy the effect grouping has on text, so I'm just going to mention it really quickly. (Also, it's probably the easiest way to explain the way the grouped layers in the above icon work.)
In order to group two layers together, hold down Alt and position the mouse arrow between the two layers you want to group. The arrow will turn into a padlock. Click, and the layers are grouped. Redo the same thing to ungroup.
Here's a little demonstration using the sky in the above stock image.
Masks can be used on all kinds of layer, text layers too.
This is really useful, because by "unlinking" the mask and then selecting the layer, you can move around the text and the mask will stay in place.
Masks are something you really only start to understand once you start using them, so it's best just to learn as you go along.
So, questions? Comments? </div>
I also capped four episodes of VM, but I haven't even started uploading zips. CotT, LotB and MvM are uploaded to vm-caps.com and RB will be done in an hour or so.