Hestercombe was first mentioned in an Anglo-Saxon charter of 854 and from 1391 until 1872 it was continuously owned by one family, the Warres.
The first reference to a garden at Hestercombe is from a medieval perambulation of 1249 which records 'my Lord of Hestercombe's garden'. The next reference is from the estate accounts of Sir Francis Warre for, ironically, pulling down a garden hedge in 1698. In 1731 John Bampfylde, MP for Exeter, who had married Sir Francis's daughter Margaret in 1718, commissioned plans for a garden from a Mr Brown of London. Coplestone Warre Bampfylde designed and laid out the Landscape garden as we know it today, after inheriting the estate from his father in 1750.
Miss Elizabeth Warre, the last of her line, died in 1872 and the following year, Hestercombe was acquired by the 1st Viscount Portman who undertook considerable remodelling of the house including the creation of the Victorian Terrace.
In 1903, the Hon Edward Portman, his grandson and Hon Mrs Portman, commissioned Sir Edwin Lutyens to create a new Formal garden
In 1944 the estate was sold to the Crown Estate who continue to hold the land apart from Hestercombe House itself and the Formal gardens. These were sold to the Somerset County Council in 1978. Since October 2003, the gardens have been managed by the Hestercombe Gardens Trust. The Somerset County Council and the Trust are in the process of transferring ownership of Hestercombe house to the Trust and so the gardens and house will once again be reunited.