5) James Madison (Net worth: $101 million)

Fourth president of the US, James Madison, held the office of the president between 1809 and 1817. He was the largest landowner in Orange County, Virginia. He made his fortune as secretary of state and president. His land holding consisted of 5,000 acres and the Montpelier Estate.