To meet the demand for young students learning computer science, computer science curricula for students in elementary school (ages 5-11) are emerging. With those curricula comes the question, how do we gauge young students' computer science knowledge and practices? This question presents many challenges, in part because no validated assessments exist for computer science in elementary school; written assessments are often not appropriate for this age group; and one-on-one assessment with teachers, a common practice for very young learners, presents challenges with respect to scalability and teacher preparation. To address these challenges, I propose to build an animated spoken dialogue agent that uses embedded assessment techniques to gather information on students' knowledge and practices. The data collected through this agent will help us understand how young children learn computer science, lead us to refine computer science curricula for young learners, and inform the development of adaptive individualized support for these students.