"mostly zigbee?" not sure what you mean there. But ESPHome can be controlled directly without HA. You should read the website, specifically the sections on "Networking" and "Management and Monitoring".

If you are starting at zero there is a big learning curve, but if you're into it, it is a lot of fun.