The church was located close to an earlier medieval church with the same dedication, which was recorded as derelict in the mid-seventeenth century, and in ruins on the six-inch OS map.

The joint dedication to Peter may have been prompted by the medieval dedication to Peter at Steynton, the parish in which Milford Haven was located.