Responding to public interest, John Pascoe Fawkner convened a meeting on 25 November 1848 that led to the formation of the Victorian Horticultural Society. Through the efforts of Charles La Trobe and Melbourne's second mayor, Henry Moor, the government Survey Paddock at Richmond was made available to the new society. The first plan for the site, prepared in 1860 by Alfred Lynch of Prahran, included exhibition buildings, conservatories and formal plantings on an axial plan with parterres reminiscent of 17th and 18th-century European gardens. Through the transfer of some of Richmond Park the site grew from 25 to 35 acres (10-14 ha) by 1865. The early focus of the Gardens was on the acclimatisation of introduced fruit to Australian conditions, developing new varieties and improving the breeding stock.
A cottage was built on the garden site to accommodate the earliest directors, Mr Phillips (1863-68), W. Clarson (1868-76) and George Neilson. In 1884 the Horticultural Society built a pavilion in the Gardens for its shows. Five years after being granted its 'Royal' charter in 1885 the society declared bankruptcy. In 1891 the site was taken over by the Department of Agriculture, which established the School of Horticulture, a name that continued until 1917. In November 1897 Charles Luffman became director, extending the residence and redesigning the Gardens along less formal lines. Women, first admitted in 1899, became a regular feature of the school.
For many years the Gardens offered a rural character within the city, with Jersey and Friesian cattle herds and a billabong adjacent to the Yarra River (the site was subject to flooding in 1863, 1878, 1891 and most seriously in 1934). This changed gradually with the completion of the Plant Research Laboratory in 1929, the construction of a new educational building (completed in 1949), and the removal of the 19th-century exhibition hall and the principal's residence in 1980.
Much of the late 19th-century form of the Gardens remains, though with extensive additional building, including a rockery by Ellis Stone, a new stream and pond system by Robert Boyle in the native garden, and a courtyard by Steve Mullany, a Californian landscape designer.