It's hydrogen inside an oxygen atmosphere that's the problem.
(So on Jupiter, you wouldn't want to ride in an oxygen balloon, ie you wouldn't want to ride in a hot air balloon there.)