Oh huh, I was always under the impression that "X" meant "Cross", as in "Cross-Desktop Group", since they were working on standardizing things across different desktop environments. But I guess the "X" was just a reference to X11? I guess today it encompasses a lot more than X11-based stuff, though...
[0] https://en.wikipedia.org/wiki/Freedesktop.org