Hiya!

There are a few ways to go about automated coastlines in PS, although none of them is quite the same than drawing them by hand. I do, however, have a method that works out reasonably well in my opinion.

How are your coastlines made? Do you have a separate layer mask for the landmass?

If you are working with a masked layer for a coastline, this is what I do:
- select the landmass by pressing down Ctrl (Windows) or Cmd (Mac) and then clicking on the layer thumbnail in the layer panel
- create a new empty layer on top of the landmass one
- select the new layer, and go to Edit > Stroke (stroke it with 1-2 pixel width, Outside or Inside of the selection can both work, matter of taste)
- when the stroke is made, use Ctrl+D to Deselect the landmass

This is the basic method. I use these additional steps that get me a pretty nice looking stroke when I don't have the time to do it by hand:
- select the stroke layer you just made (the same way you did with the landmass earlier)
- create a new layer
- with the Lasso or Marquee tool, right-click on anywhere in the working area. Pick Make work path... and set the Tolerance to something around 0,5-1,5.
- go to the Paths panel
- right click on the Work Path you just made, and select Fill.

Now you should have a coastline, that is a bit uneven to give it more of a hand-drawn feeling. You can merge the two line layers into one.

This only works if you have a relatively cleanly made landmass layer/mask, however. So not knowing how you have made your coastlines, I don't know if this helps you or not. If not, maybe include some screen caps of what your layers look like and I can try to help you further.