Designed in 1773 by David Henderson, Inverleith House was built in 1774 for Sir James Rocheid (1715-1787) as his family home and the centre-piece of his estate at a total cost of £4,109.
Around 1820 a part of the Inverleith estate was sold to become the new location for the Royal Botanic Garden Edinburgh (RBGE). The house was gifted to the Crown in 1877 together with the surrounding land for the purpose of extending the activities of RBGE and for the enjoyment of the public. After restoration work following a fire, Inverleith House became the official residence of the Regius Keeper (Director) of RBGE.
From 1960 to 1984, it housed the new Scottish National Gallery of Modern Art, until it moved to it's current site in the building of the former John Watson's Institution in west Edinburgh.
On 4 December 1970 the house was listed as a historic building in category B, with Historic Scotland id 28081. An extension was built in 1974. Since 1986 Inverleith House has been used as exhibition space by RBGE.