The short-lived effort shows how Japan’s biggest brokerage continued high-risk bets even as executives say they’ve overhauled to focused on serving customers.