“A loud expression of that desire was heard during the Polk administration.”