Woodbury's first library was established in 1772, but was in existence for only a few years. Today's library has its roots in the Woodbury Library Association which began in 1851 with a collection of 200 books and was located on the second floor of the Town Hall. On June 14, 1897 all assets of the Woodbury Library Association were transferred to the Town of Woodbury.
After 1902 the library was housed in the former Parker Academy building. At first the library only occupied the first floor, but by the 1960's had occupied all three floors, with the children's floor in the basement, and had even begun lending records. The Parker Academy building was lowered in 1980, when thecurrent library was constructed. The new building brought many changes, but the most important was having a place for patrons to sit. The buildings underwent renovations in 1997.