Le musée d'Orsay - the museum of Orsay (from a place name, no preceding article) - "de Orsay"  becomes « d'Orsay » by elision.

Le musée du Louvre - the museum of the Louvre  (le Grand Louvre) - "de le"  becomes « du » by contraction.

Thanks Maarten. But is Louvre a place name as well, thus no preceding article? Orsay and Louvre are both place name, nope?

The Louvre is now considered as a 'place' in everyday speech. Its name however is  « Le (Grand) Louvre ». For places that have an article as part of their name - Le Havre/La Rochelle - contraction occurs where appropriate eg au Havre/du Havre, or we use à La/de La Rochelle where contraction is not appropriate. 

However, most cities/local place names do not have an article eg Paris/Londres so we get à Paris/de Londres. Hence with Orsay - no article results in d(e)'Orsay

À = To/in and De = From/of with cities in French (French Prepositions of Location)

Because theres a vowel. (when a vowel appears it will become d(e)'Orsay ) because you omit the e. 

Pansiluni, that is not the answer to the question though, which relates to the fact that ‘Orsay’ is not preceded by a definite article, whereas ‘le Louvre’ is.  If it was ‘L’Orsay’ it would then have become “de l’Orsay”, but it’s not, and it doesn’t.