Documents refer to the existence of a castle in this place since the 10th century, but the present castle is most probably a reconstruction of the previous fortress, carried out by the Visconti family when they took possession of Brivio in 1373. It was reinforced during the Sforza period as part of the defensive upgrade of the fortifications along the borders of the State of Milan promoted by Gian Galeazzo Sforza. The extension is the work of the military architect Bartolomeo Gadio. Originally, it probably had a quadrangular plan with a high tower and cylindrical corner towers.