The rue de Lille was once the central lane of the garden belonging to Henri IV's famous queen, Marguerite de Valois. On her death in 1615, the property was sold by lots, and private mansions continued to build up the neighbourhood, while on the banks of the Seine a port known as the Grenouillière served as a resting place for lumber barges and other cargo. The construction of the Quai d'Orsay began in 1708 near the Pont Royal, and was completed a century later under Napoleon I's Empire. The aristocratic vocation of the neighbourhood was already well established at the end of the 18th century, when the Hôtel de Salm (today the Musée de la Légion d'honneur) was built, between 1782 and 1788.
During the 19th century, two buildings stood upon the site of the future Orsay station: the Cavalry barracks and the Palais d'Orsay, built between 1810 and 1838 successively by Jean-Charles Bonnard and Jacques Lacornée. Although the Palais had originally been planned for the Ministry of Foreign Affairs, it eventually housed the Cour des Comptes (Court of Accounts) and the Conseil d'Etat (State Council). During the violent upheaval known as the Paris Commune in 1871, the entire neighbourhood was burnt down. For thirty years, the ruins of the Palais d'Orsay served as reminders of the horrors of civil war.