When put face down in this way, cards represent lands: face down red cards are MINES, face down green cards are FORESTS, face down blue cards are RIVERS, and face down white cards are PASTURES.
These four lands are considered to be ADJACENT to your Castle but not to each other: until you get a Stables, adjacent in Barons! means edge-adjacent.
As will soon be indicated by the Castle card itself, place the cards in the following manner: pasture (white) on the top, river (blue) to the right, forest (green) on the bottom, and mine (red) to the left.