Located in St Kilda Road, the Victoria Barracks were constructed from 1856 to house British troops who had earlier occupied temporary camps at Flagstaff Hill, Batmans Hill and Princes Bridge. In 1854 the British military headquarters moved from Sydney to Melbourne, reflecting Victoria's importance consequent on the gold discoveries, and in 1859 a military reserve of 168 acres (67.2 ha) was set aside. From 1863 the Barracks also housed the administrative headquarters of both the Victorian and imperial military forces, and became known as Victoria Barracks from 1866.
After British forces were withdrawn from the colonies in 1870, the Barracks housed the headquarters of Victoria's naval and military forces, first as the home of a permanent local artillery unit. Other functions in the 1870s included accommodation for Industrial School children, indigent aged men from the Immigrant Aid Society, and the Victorian Police Depot. During the New Zealand and South African wars, and the Boxer rebellion in China, Barracks staff administered the recruitment, dispatch and occasionally the quartering of the volunteers. It was here also that Colonel Tom Price gave his infamous order to the Victorian Mounted Rifles during the maritime strike in 1891 to 'Fire low and lay them out', though ultimately the dockside situation was handled by the police.
By the time of Federation in 1901, the reserve had been reduced to 12 acres (4.8 ha), enclosing established buildings and the square. With Melbourne appointed the temporary seat of the Commonwealth government, the Minister of Defence John Forrest established his offices at Victoria Barracks, and from 1902 until the Military Board of Administration was formed in 1905, the Barracks became the military headquarters of the Commonwealth Defence Forces under Major-General E.T. Hutton as commander-in-chief Australian Military Forces. During World War I the formation and dispatch of Victorian AIF units was managed from the Barracks, most of the troop accommodation being converted into offices for the expanding administrative staff.
Although the national seat of government moved to Canberra in 1927, services headquarters continued in Melbourne, and with the coming of World War II in 1939 Victoria Barracks became the site at which most of the important decisions were taken. The Advisory War Council met in the War Room from 27 September 1939. From 18 April 1942 all combat units of the Australian armed forces were assigned to US General Douglas MacArthur's command. Two days later there assembled the first meeting of the War Conference set up by Prime Minister John Curtin for the higher direction of the war. Japanese submarines off the coast and at least one overflight of Melbourne by Japanese aircraft brought the threat of attack, and Barracks staff became very active in the defence of Victoria's coastline.
In the postwar years Victoria Barracks reverted to its role as the administrative headquarters for army units in Australia. By 1958 the Command units of the Defence Department had moved to Canberra. Some Defence offices remained in the Barracks until they were relocated to the new Defence Centre at 661 Bourke Street. Significant war trophies contribute to the imposing face that Victoria Barracks presents to St Kilda Road: two muzzle-loading guns captured from the Russians in 1854 during the Crimean War, and two German guns captured in World War I during 1918, one taken on the Western Front, the other in Palestine.