Japan’s space agency said its H3 rocket carrying a navigation satellite failed to put the payload into a planned orbit, a setback for the country’s new flagship rocket and its space launch program.