His first appearance as conductor was owing to an accident: in 1924 the orchestra's conductor fell ill and Ormándy had to take his place. After a successful debut, he conducted more and more often, so when the talkies made the cinema orchestra redundant, Ormándy sought work as a conductor. He conducted popular classics on radio and in the summer undertook to conduct various orchestral performances. By the summer of 1930 he appeared in the summer concerts of the Philharmonic.
His big break as a conductor came in 1931, when, because of the illness of Arturo Toscanini, he was able to conduct three performances of the Minneapolis Symphony Orchestra. His unfailingly sure touch as a musician, his radiating personality and his high standards regarding orchestral sound so impressed the orchestra management, that in the same year they signed a contract with him. He spent five years with the Minneapolis orchestra, during which time they recorded for RCA Victor, besides symphonies by Mozart, Beethoven and Mahler, as well as orchestral works by de Falla, Sibelius and Rachmaninov, the first recordings of Schönberg's Verklarte Nacht and Kodály's Háry Suite. Under his direction, the orchestra became one of the leading ensembles in America and the world. Nonetheless, later on Ormándy's name became linked not with the Minneapolis Orchestra, but with the Philadelphia Symphony. In 1936 Leopold Stokowski offered him the post of second conductor of the ensemble. After Stokowski's departure, Ormándy led the Philadelphia Symphony Orchestra for 35 years, till 1973. As the recordings made for Columbia demonstrate, it was Ormándy who created the ideal modern orchestral sound based on the perfect homogeneity of instrumental playing, technical perfection, and a bright, intense sound, which to this day serves as a model for the leading orchestras of the world.
Ormándy's repertoire concentrated primarily on the works of the late romantic period and the first decades of the twentieth century, but in his concerts and recordings he popularised, besides the classics, contemporary composers as well. The original premiere of Bartók's Third Piano Concerto, Rachmaninov's Symphonic Dances, and orchestral works by Kodály, Britten, Copland, Barber, Shostakovich and others is connected with his name. His prestige is indicated by the fact that in 1948 he conducted the first ever television concert in America. Though Ormándy regularly toured Western Europe from 1949 onwards, both with his orchestra and as a guest conductor, and kept closely in touch with his Hungarian musician friends – particularly with his one-time harmonics teacher, Zoltán Kodály – he did not give concerts in Hungary after the Second World War. The Budapest audience last met him in 1936, when at the festive concert held on the 250th anniversary of retaking Buda Castle from the Turks, he conducted the Budapest Concert Orchestra in works by Liszt, Erkel, Hubay, Dohnányi, Bartók and Kodály.