Except that is bullshit. They did worked the other parts of the year too, just not doing the exact same agricultural work as those few weeks.
That thing simply ignores everything it takes to keep animals alive year round, keep kids alive year round, create and repair tools, keep house warm, create fabric, sew cloth, actually cook without modern tools and so on and so forth.
Just because there is a rush time does not mean workers do nothing the rest od the time.
Yes. In the middle ages (and presumably in any agrarian society) people would work intensely for a few weeks and have the most of the year free.