People have done this. See, e.g. Brouwer and Heeger (2009), Decoding and Reconstructing Color from Responses in Human Visual Cortex.