This study investigates the differing experiences of buildings between architects and laypersons, focusing on the physical and cognitive foundations of these differences. Both groups assessed the global aesthetic quality and six cognitive properties—complexity, clarity, friendliness, originality, meaningfulness, and ruggedness—of 42 large contemporary buildings. Additionally, 59 physical features of each building were independently evaluated. Lens model analyses revealed that the two groups interpret physical features differently, leading to varied cognitive perceptions and, consequently, distinct aesthetic judgments.