Using Vector Mask for best quality
You must have noticed that when you use layer mask, it creates a soft edged mask around the selected area of the object. Later when you transform such masked area, you will get jagged edged object. The quality of the image degrades if such image is transformed or converted into higher resolution later on. We can avoid this by using vector mask. Make a selection using a path or take an object created using shape tool. Then convert the path or shape into vector mask (Layer > Vector Mask > Current Path). Like layer mask, vector mask is only applied to layers not to backgrounds. So, now if you transform the image to higher sizes, you won’t get jagged edge.