Project Orion, or Project Bang-Bang as it was more familiarly called, had a lot more issues than that.

The minimum unit of propulsion was one fission bomb, and there's a decided limit to how small you can make one.

More, you get all the added momentum in a millisecond or so, and any moderately sized rocket would experience 100s of gs of accelerate and have its contents destroyed, so you need to fly some immensely huge thing about the size of Long Island, and be setting off tiny atom bombs at the stern continuously.

And what exactly is the substance at the stern that protects the rest of the ship from the radiation of atom bombs continuously exploding? A clear case for unobtanium.