The Portuguese-Jewish family De Pinto took up residence here in 1651, just outside the actual Jewish neighbourhood. The most well-known resident, Izaäc de Pinto (1717-1787), was the chief administrator of both the East and the West Indies Company. His initials can be found in the wrought iron trelliswork in front of the windows.
The premises threatened to be demolished around 1970 to make room for the underground. Protest groups arose and they managed to convince the municipality to spare the historical monument. The building was subsequently restored in 1975 and functioned as a public library untill 2012. Citizens in the neighbourhood have since worked to keep the building open for the public, as a cultural-literary meeting place.
Each year a multitude of exhibitions are held throughout Huis De Pinto. The exhibitions are accompanied by a program of debates and concerts.