Reviewer Guidelines

Preamble

The LAK conference involves a rigorous review process with two purposes:

• Supporting fair and informed acceptance decisions based on robust assessments of submission quality
• Providing authors with useful feedback about the research they have conducted and their communication of it to the interdisciplinary learning analytics community

 

Guidelines

To meet this dual purpose the following guidelines are provided to support the community in providing high quality reviews. The creation of these guidelines was informed by other review guidelines used by related journals and conferences in the field such as the Journal of Learning Analytics (JLA) and the Educational Data Mining (EDM) conference. 

 

Reviews should include:

• A total of 200-500 words of detailed feedback that give a complete and rationaled assessment of a submission (the kind you would like to receive for your own work). Please avoid very short reviews, they are frustrating for authors and detrimental to the overall review process. 
• The review should include:
  • A brief summary of the paper itself (e.g., the question being addressed, the high level approach utilised, what was found).
  • A thorough assessment of the submission’s main strengths and weaknesses in making a substantial conceptual, technical or empirical contribution to learning analytics. 
  • Where possible, suggestions for improvement should be given.
• The following categories and questions are useful to consider in writing your review
  • Relevance: 
    • Is the submission trying to solve an important educational problem related to the design, development, implementation or evaluation of learning analytics?
    • Does the submission attend to the real-world context, including issues of impact, fairness and equity?
  • Novelty: 
    • Is there a novel contribution in the submission in relation to previous work in the area?
    • If a replication study is reported, is it clear what is the contribution to knowledge in comparison to the original study?
  • Grounding:  
    • Is the work situated appropriately with respect to the current state of the field, including sufficient coverage of relevant literature and current theories of learning?
  • Methods: 
    • Are the methods used suitable, well-described and justified with reference to the literature?
    • Does the submission show accepted evidence of rigour in the tradition followed (statistical, computational, qualitative, design)?
  • Results: 
    • Do the claims made have appropriate empirical support?
    • If negative results are presented, have different explanations for the lack of findings been considered?

• • Implications
    • Are contributions to theory and/or practice outlined clearly? 
    • Are limitations with respect to data, analysis or framing factors taken into account? 
    • Are potential issues of fairness and equity considered?
  • Communication: 
    • Is the submission written clearly for understanding by an interdisciplinary audience?

Numeric Assessment

• In addition to the textual review, you should provide two numeric assessments of each submission that align with the textual comments you have made:
  • A 7-point scale numeric ranking {-3 to 3} indicative of the overall evaluation of the quality of the paper that must be supported by your written appraisal
  • A 5-point scale numeric ranking {-2 to 2} indicative of your confidence in assessing work in the area addressed by the paper

Additional Considerations

1. If the paper is not properly blinded (i.e. the identity of the authors is revealed in the paper) please review the paper as usual, but indicate this in the “Confidential remarks for the program committee” box on the reviewing form.
2. If there are issues with the English in the submission (e.g. grammatical mistakes, misspellings or unusual phrasings), this can be noted in the “Confidential remarks for the program committee” box on the reviewing form, but it should not affect the review and assessment of the submissions with respect to its scientific merit.
3. Meta-reviewers will compare all reviews and numeric assessments of quality and confidence as well as author rebuttals (if made) and make final recommendations for paper acceptance or rejection with justification to the program committee chairs.