A ninth-century charter, found in a sixteenth-century manuscript, refers to the church as Abbas Llan Sanfrigt, which would be an exceptionally early attestation of the dedication of a church in Wales.

The present church was rebuilt in 1868 and retains medieval features. However, there is evidence of an earlier church site, slightly to the north-west and now located on the shore. According to Samuel Lewis, writing in the early nineteenth century, the chapel was used as a salting-house for curing fish, and stone coffins were known to have been washed away.