What you're describing can be caused by the selection clipping what would be a smoothed area. If you feather the selection a bit (say, 0.5 to 1.0) before doing the precipiton erosion step, it will get an interesting bathtub ring. If you invert the selection and set the sea to a small negative value (-1 to -10), the landscape will get cut back from selection somewhat and you'll want a height clip operation from 0.01 to 1000000000 in order to force things back to the original selection.

It can also be caused by how the precipiton algorithm works if you've only got a little ledge between a high and a low place. The algorithm moves Delta amount of the difference between the high and low terrain in the downhill direction. For example, a sample at 10 and a sample at 1010 will move Delta*(1010-10) altitude onto the 10 pixel. With a default Delta of 0.25, that would be a change in altitudes from (10, 1010) to (260, 760). This process continues moving altitude downhill. Small altitudes will get overwhelmed pretty quickly if they are next to a large one. Transferred altitude that goes outside of the selection (or off the map edge) just disappears. The default Delta of 0.25 is the largest value that will do the right thing. Using a smaller Delta will probably give better results in this case, but it will take longer to get similar results.