Both are problems to solve, architecture is as much of a problem as data.

Data has the problem of getting successively tainted by LLMs as well as the lack of open high-quality datasets.

While architecture has the problem of shifting too much focus on a flawed architecture - transformers.