PSP stamp edges, tested on PSP 7. It's not perfect and it's not one-step, but you could try this for a
picture-frame style overlay:
Select a rectangle shape then under line style select 'custom'
Check the box for different segment caps and select all your caps to be 'ball', set your ball size (maximum is 10 I think?) then set your segment and gap lengths until they look stamp hole like using the sliders and number box. Save with a new name.
On a new layer, draw a rectangle as line only, no fill, with the new line style.
Use the magic wand with opacity set to select the area outside the stamp, expand the selection a pixel or two if needed, then fill the outside.
Alpha channel mask variation:
Create the line style, then set the fill the same as a line and draw your blank stamp. Once created, you can use this layer to create a mask and save this to the alpha channel to apply to any other layer(s).
Now you have made your custom line you can use it on rectangles of different sizes, and if you create your rectangles as vectors you can resize them as needed.
No idea how this would work on the latest versions of PSP, I'm afraid.
