In The Life and Letters of John Hay (William Roscoe Thayer, 1916) the quote is given but not referenced to a specific letter. Another source says this comes from a letter to Nora Perry in 1859.