Selecting random effects in a linear mixed model when all participants complete all repeated measures?

Question

Participants read 10 pairs of stories from Author A and Author B on 10 different topics (different topic for each pair). For each pair, they indicated whether they preferred Author A, Author B, or neither according to 10 criteria. All participants completed all criteria ratings for all prompts so they made 10x10 = 100 ratings total.

I'd like to understand which Author participants preferred, both overall and for each criterion. I was thinking of using a linear mixed model with the DV veing individual criterion ratings or average rating across criteria for each pair. I have two concerns.

1. With only a 3-point scale, will this violate any assumptions like normality?

2. Should I add participant ID and prompt/pair version as random effects (or would the prompt/pair be a fixed effect)? If all participants complete all pairs once, and only once, is the model able to parse variation due to the participant and due to prompt/pair?

Answer 1

1. Yes, you should be concerned about non-normality with only a 3 point scale. If you have enough participants it may not matter, but you may want to look into something like Proportional Odds Logistic Regression instead of normal linear models (though I do not know of any software that handles POLR models with random effects, but have done permutation tests to handle them).

2. Whether to include participants as fixed effects or random effects depends on what level you want to infer your results to. If you care about your specific participants, but nobody else, then include them as fixed effects. If you consider the participants in your study as representative of a larger group that you want to make inference about (which author would someone not in your study likely prefer) then they should be random effects.