19 Sept. 1799–27 Dec. 1868
After several trips to Florida and Alabama, Whitfield in 1835 moved his family and the people he enslaved to Marengo County, Alabama, where he acquired large tracts of land. Eight years later he purchased from George S. Gaines a farm that he named Marlmont near the town of Demopolis. There he located his household and began constructing a sizable residence.
By 1850, Whitfield was the enslaver of 167 people, and owned real estate valued at $53,000. The new house took on even more substantial proportions as Whitfield, an accomplished amateur artist, architect, and engineer, designed more and more rooms. By about 1860, Gaineswood, as he called the mansion, was a showplace. During the preceding decade Whitfield had assisted in building an Episcopal church and a plank road and had increased his fortune to $142,000 in real estate and $300,000 in personal property (most of which was the value of the people he enslaved).
Whitfield married Elizabeth Watkins in February 1819, and they had twelve children: Sarah Watkins, Winifred Bryan, Nathan Bryan, Jr., Nathan Bryan III, Mary Elizabeth, Bryan Watkins, Needham George, Edith Winifred, James Bryan, Sarah Elizabeth Watkins, Edith James, and Betsy Winifred. His first wife having died on 4 Nov. 1846, Whitfield on 26 July 1857 married Bettie of Baltimore, and she bore him one daughter, Natalie Ashe.
The completion of Gaineswood and the period of Whitfield's greatest wealth coincided with the outbreak of the Civil War. Both his fortune and health suffered during the conflict; he died three years after it ended and was buried in Riverside Cemetery, Demopolis. In the following decades Gaineswood was neglected as it passed through several ownerships. Recognizing the structure as architecturally the most important mansion in the state, the Alabama Historical Commission in 1971 undertook its restoration. It is now a National Historic Landmark and open to the public.