Borrowers are increasingly swapping loans they received from direct lenders for more affordable debt from Wall Street banks.