Hermitage is in the heart of the downs three-quarters of a mile south of the M4 motorway close to the junction with the A34. Like several other villages in this part of the Berkshire Downs, Hermitage is a "dormitory" village of mainly 20th century housing and there appears to be no recognisable village centre. There are a number of back lanes however and here and there dotted around the village a few period houses can be found.

The parish church is the Church of the Holy Trinity, a 19th century whitewashed building which was originally known as the 'Adelaide Chapel' after Queen Adelaide, William IV's queen. It was built by the vicar of Hampstead Norreys who was personal chaplain to the queen who took a great interest in the building and contributed to the overall cost.

To the south-west of the village is Denison Barracks which was used as an American military hospital during the Second World War and became the home of the Royal School of Military Survey in 1949. The Military Stabilisation Support Unit, the Defence Cultural Specialist Unit, Land Intelligence Fusion Centre and 15 Psychological Operations Group currently use the site.

To the south of the village in Grimsbury Wood is Grimsbury Castle, comprising a large circular 'multiple enclosure' encampment which was occupied at least between the third and second centuries BC. There is an 18th-century folly on the site, also known as Grimsbury Castle.

The village got its name because there was reputed to have been a hermit living in the nearby woods near a well which was said to have curative properties.

Hermitage is about five miles north-east of Newbury on the B4009.