Acton's population is largely white, with only a few non-white residents. Although a growing community, the population is still disproportionately white, with almost seventy percent of residents being white. The Census Bureau defines non-white residents as Hispanic or Latino, which describes people of Spanish heritage. Although the census data is only three years old, Acton's population has been stable over the past decade.
The town's 1820 census reports that only seventeen free colored people lived in Acton, and the record is incomplete. However, John Oliver's two-person household was listed as white. Acton's black residents were concentrated in East and North Acton. The Olivers' household included fifteen black cattle, including three milking cows. Joel Oliver's family eventually moved to East Acton, and his father may have helped finance his move.
During the 1800s, students in Acton City needed to seek opportunities outside of school. Young men tended to seek higher education from learned individuals outside the schoolhouse, including the town's minister. Female students also needed to find other opportunities, and many awaited higher education. Many boarded at private academies or commuted to nearby high schools. While this was not an option for most Acton residents, there were still opportunities for both sexes to gain an education.
In the nineteenth century, the town was growing, with the railroad bringing new industry and people. The railroad also brought new immigrants to the area. In addition to this, the town became a rail hub and served as a division point for the Marlborough Branch Railroad. Two other railroads crossed Acton in 1892, forming the Fitchburg Railway. Acton and Boxborough shared a double track right of way, but split in North Acton.