Frilsham is a small village in the southern, wooded part of the Berkshire Downs. The main part of the village comprises mainly 20th century houses and there is no evidence of an older village. It is sited on high ground surrounded by woods and meadows like much of this part of the Berkshire Downs. The village was originally around the church at the southern end of the parish and consisted mainly of tenant farmers.

Frilsham manor house and the parish church, dedicated to St. Frideswide, are three quarters of a mile to the west of the village on the bank of the River Pang. The church building is essentially Norman, though probably built on the site of an older Saxon church. The chancel is 15th century and the brick tower and porch were added in the 1830s. The churchyard, well known for its snowdrops and winter aconites, looks as if it was originally circular, suggesting a much older pagan holy site. St. Frideswide was the Saxon patron saint of Oxford.

There is a local legend that St. Frideswide, fleeing from the unwanted attentions of a Mercian prince in the late 7th century came to Frilsham and concealed herself as a swineherd. It is even suggested that her pig sty is on the site of the current church.

However, the reputed site of the well where she is said to have healed local people is at the top of the hill, on Frilsham common. St. Frideswide may have hidden there for some time, sustained by a nearby well, which is preserved to this day in the wooded common. 

Frilsham is half a mile south of the M4 motorway, 4 miles north-east of Newbury.