The history of Paris starts in the Île de la Cité: the first inhabitants of Lutèce may have lived on this island, before the Romans developed it together with the Left Bank (“Latin Quarter”), and later, medieval kings the Right Bank.
Nowadays, it’s difficult to imagine that the Cité is the oldest part of Paris: Haussmann’s renovation of Paris destroyed the small streets which testified to its historical role and homogeneity (refer to this page), and the omnipresence of police and justice institutions break most of its charm. Therefore, most tourists only visit the two big spots of the island: Notre-Dame cathedral and Sainte-Chapelle, famous for its stained glass windows.
But thankfully, two areas have been slightly preserved. In the West, the deserted and quiet Place Dauphine (see the Place Dauphine article) is one of the most charming squares in Paris. And in the East, between the northern arm of the Seine and Notre-Dame, a few streets have kept a sort of medieval romantic character. They are generally ignored by the crowds, which gives them even more quietness and mystery, especially in the evening onwards.
So, if you like wandering off the beaten track, explore narrow Rue Chanoinesse, and walk into neighbouring Rues de la Colombe, des Ursins, des Chantres, and Massillon, onto the steps of Esmeralda and Quasimodo. You might be overwhelmed by the contrast between these streets where time seems to have stopped and the big touristic confusion nearby at Notre-Dame, just one block away!