We have seen a dramatic rise in the use of computational devices/systems for our information and entertainment needs. The growing reliance on these technologies has motivated researchers to work on the development of “natural” user interfaces that can understand the intent and preferences of users and are easier to use. Digital assistants (Siri/Cortana), Search engines (Google/Bing), and Gaming consoles (Kinect/Nintendo Wii) are some popular examples. Making machines understand human intentions and preferences is an exceptionally challenging problem. Part of this difficulty lies in capturing the large variability in the preferences and behaviour of different users. Using examples from our work on personalized recommendation systems and Kinect-based gesture recognition interfaces, I will discuss the challenges of developing a system that is supposed to work on “everybody”.