The South Asheville Cemetery began as a slave burial ground, and its first known caretaker was an enslaved person named George Avery (1844-1938). Mr. Avery was owned by William Wallace McDowell (1823-1893), who lived in the Smith-McDowell House, and Mr. McDowell entrusted Mr. Avery as the manager of this cemetery, located on the family's property. Recognizing that the Confederacy was going to lose the Civil War, Mr. Avery decided to join the Union Army, and after the war Mr. Avery earned a pension as a result of his service in Company D, 40th United States Colored Troops. Upon his return to Buncombe County, Mr. Avery continued to oversee burials at the cemetery until his death in 1938, though he left no written burial records about the cemetery or its occupants. Mr. Avery's monument is one of only ninety-three headstones that have names or dates identifying the people buried at this site, but the South Asheville Cemetery is a two-acre burial ground that serves as the final resting place for approximately two thousand African Americans.

During the 20th century, the neighborhood surrounding the cemetery would come to be called South Asheville. This area was absorbed into Kenilworth and then, subsequently, into the City of Asheville. African American residents of South Asheville mostly attended two churches, St. John “A” Baptist and St. Mark A.M.E. Church. Over this same time period, the South Asheville Cemetery was one of only a few cemeteries for African Americans in the region, and it is notably the oldest public African American cemetery in western North Carolina.

Part of the South Asheville Cemetery was allotted for church congregants, but any African American community member could be buried in the cemetery for a nominal fee. Many of these people were buried in wicker baskets or pine coffins, their graves marked only by field stones or handmade crosses. Due to the settling of the ground and the array of unusual grave markers, the cemetery must be cleared by hand. The South Asheville Cemetery was closed after the City of Asheville annexed South Asheville and Kenilworth, and the last person interred there was Robert C. Watkins, buried in 1943.

The South Asheville Cemetery fell into disrepair during the mid-twentieth century, but in the 1980s members of the St. John "A" Baptist Church community--most notably George Gibson and George Taylor--began restoration efforts on the property. It was brought back to the public’s attention over this time period when a series of oral history recordings, now housed at the UNC-Asheville Special Collections Library, documented people’s recollections of the cemetery. Over the last thirty years, thousands of volunteers have worked with members of the South Asheville Cemetery Association to improve and maintain this sacred and historical site in an effort to promote greater public awareness of African American history in Buncombe County and to honor the people buried there.