There is an excellent article over at the American Presidents Blog on Dolly and James Madison as seen through the eyes of a former slave.
Paul Jennings, in his memoir, describes Dolly as
“a remarkably fine woman She was beloved by every body in Washington, white and colored. Whenever soldiers marched by, during the war, she always sent out and invited them in to win and refreshments, giving them liverally of the best in the house.”
James, he says, is simply “is one of the best men who ever lived.” He goes on to commend his even-temperedness, his moderation, as well as his treatment of slaves.
Contrast these with the words of Washington Irving who described Mrs. Madison as a “fine, portly, buxom dame” and her husband a “withered little applejohn”.
Sounds like someone woke up on the wrong side of the bed, Mr. Irving. For shame!