Waller was born in Somerville, Fayette County, Tennessee, USA, on 1 December 1848. She grew up in East Tennessee, except during the Civil War, when she was attending school in Corinth, Mississippi. She had five siblings.

On 2 December 1875, at the age of 27, she married Dempsey Thomas Waller (1833–1890) in the old Peabody Hotel in Memphis, where Lowenstein House currently is. The couple had three daughters, including Luxora Waller (1879–1959). She named her daughter after a character from a book she greatly admired, titled “Pillar of Fire.” Before marriage, she resided close to Somerville. However, as her husband was a young business pioneer just across the border in Arkansas, they decided to relocate there. Her husband purchased a substantial piece of land a few miles north of Osceola and constructed a one-room dwelling that also served as a store near a boat landing. A couple of years later, when the settlement required a name, her husband proposed naming it after their daughter, Luxora. Her husband passed away in 1890

Waller remained an active gardener until she reached the age of 100. When asked about the secret of her longevity, she attributed it to a “combination of faith in mankind and hope for the future through memories of the past.”

Waller passed away in Luxora, Mississippi County, Arkansas, USA, on 4 January 1959, at the age of 110 years, 34 days.