My initial observations of some the issues are how LA, which appears to have the purpose of modeling an individual's performance and then taking some action based on that model, handles complex statistical procedures. Not the most exciting topic for a blog entry, so I'll be brief.
1. What is the process for model selection in an LA model that encompasses a variety of variables? One question that is important to address is the process of model selection. That is, when researchers develop a prediction model, we use a procedure to select the variables that become part of the model. This procedure can be an automated procedure in a software package (such as stepwise model selection) or a deliberate procedure of testing one variable at a time (by hand) to determine the first variable in the model, the second variable, and so on until you have the final model. In LA, how does this happen? Is one model applied to the data regardless of the quality of fit? Is it transparent?
2. We must be careful to avoid saying that prediction models predict an individual's exact performance. There is always uncertainty. Statistics--especially inferential statistics--is a science that embraces error and uncertainty. That is, a prediction model always has some degree of error--and even a good model predicts the average response of some outcome variable. If I say that my prediction model indicates that this student should receive the GPA of 2.9, then I'm assuming that every student is going to have the average outcome. Some will get higher GPAs, some will get lower. If we do LA, we need to make a commitment to helping our colleagues understand the nature of error.
3. In LA from automated systems like LMS's can we be sure of the data we're analyzing? It is tempting to assume that a page view in a content system means that students read that page. Perhaps they did. As researchers, we may need to develop research methods that enable us to learn about the quality of the interaction in a system like an LMS. This can be done with some kind of instrument that asks students more about how they interacted with some electronic resource. Surely, some systems are catching up to this by enabling more interactivity (such as e-textbooks that allow highlighting, note-taking, and sharing of questions with instructors), but we should be skeptical of each variable we consider in a LA procedure.
