No you can't. Turning off Handoff turns off everything that synchronizes between your devices, not just the clipboard. For example call and imessage forwarding.

They decided to do it Gnome style and give the user no options.