The second person pronoun {\em you} serves different functions in English. Each of these different types often corresponds to a different term when translated into another language. Correctly identifying different types of {\em you} can be beneficial to machine translation systems. To address this issue, we investigate disambiguation of different types of {\em you} occurrences in multiparty meetings with a new focus on the role of hand gesture. Our empirical results have shown that incorporation of gesture improves performance on differentiating between the generic use of {\em you} (e.g., refer to people in general) and the referential use of {\em you} (e.g., refer to a specific person or a group of people). Incorporation of gesture can also compensate for limitations in automated language processing (e.g., reliable recognition of dialogue acts) and achieve comparable results.