The 27,443 square foot Thomas Colt House was built in 1865, two years after the prominent judge launched a prosperous mill in Coltsville. In the early twentieth century the building headquartered the Red Cross, then served as the site of Miss Mills School for girls. Simon England, Sr. of England Brothers bought it in 1937 and deeded it to the Women's Club, who operated there from that time until 2011.