Do TV and gaming predict children's psychosocial adjustment?

[KLSymph]

Do television and electronic games predict children's psychosocial adjustment? Longitudinal research using the UK Millennium Cohort Study

While wandering Youtube, I found this 2013 article being cited as scientific proof that video games do not cause developmental problems in kids.  It's available free for you to read, here.

If you don't read scientific papers, this should be interesting enough to merit a glance.  Test your scientific literacy!

Spoiler: ShowHide

This study's analysis is... not great.

[alethiophile]

I must say I'm suspicious of the complicated statistical analysis and the enormous numbers of covariates. The more degrees of freedom you have in your methodology, the more likely you are to produce confirmation bias. Just a plain correlation would be more interesting.

[KLSymph]

At least using regression to adjust for confounders is a legitimate thing (though I think it's overused in general, and prefer stratified sampling).

Instead, I'm talking about...

Outcome measures
Psychosocial adjustment was reported by mothers at ages 5 years and 7 years using the Strengths and Difficulties Questionnaire (SDQ), a widely-used survey instrument with high validity and reliability. The SDQ contains five scales, measuring conduct problems, emotional symptoms, inattention/
hyperactivity, peer relationship problems and prosocial behaviour. Each scale contains five items scored from 0 to 2, giving a scale range of 0 to 10. Change scores were calculated by subtracting age 5 years from age 7 years scores, to give measures ranging from -10 to +10.

ಠ_ಠ

No part of this makes sense.

[Ergoemos]

Well, I found one study that tends to agree with the paper about this "SDQ" as a useful indicator of "something" in children.

" This body supports the validity and reliability of its versions for the parent, teacher and self-reporting purpose, despite some variation in cut-off scores. The Strengths and Difficulties Questionnaire is increasingly being used as a measure of child psychopathology in other types of research, that is, aetiological, longitudinal and service evaluation studies. "

That gives them the ability to say the SDQ has High Validity and reliability, since at least one other article claimed something similar.

Now, as far as I can tell, the SDQ is measuring only one thing. The delta, or change, between a five year old's social and emotional state (as measured by the parent's opinion)  between age 5 and 7.

As far as I can tell, the number measures how much a child progresses between these two ages. A child with a measure of "5" at age five and "10" at seven will have an SDQ of +5. A second child with a measure of "0" at age five and "5" at age seven will also have a score of "+5"

There is no actual measure of a child's degree of progress, just what the progress was.

I am assuming that getting an initial score of "2" in any of the subcatagories (like hyperactivity or prosocial  problems) means the kid has more issues in that category.

A child with a score of "10" at age 5 is hyper active, doesn't get along well with peers and doesn't deal well with rules or their own emotions.

A child with a "0", likewise, has no problems.

At that point, the score makes a little more sense. Children with a score above "0" are children with more issues at age seven than they did at age five. Children whose attitudes got better will have a negative number, as their SDQ values at age five will be higher than their SDQ at age seven.

Some examples:
SDQ at age five: 0
SDQ at age seven: 10
Result: (10 - 0)= 10
Evaluation: This child's SDQ has gotten worse between the age of five and ten.

SDQ at age five: 10
SDQ at age seven: 10
Result: 10- 10 = 0
Evaluation: This child's SDQ has made little change between the age of five and ten

SDQ at age five: 0
SDQ at age seven: 0
Result: 0-0= 0
Evaluation: This child's SDQ has made little change between the age of five and ten

SDQ at age five: 10
SDQ at age seven: 0
Result: 0 - 10 = -10
Evaluation: This child's SDQ has gotten significantly better between the age of five and ten

I don't know how merit-able this method is in measuring progress, but it seems reasonable in a broad perspective, assuming the parents are generally more right than wrong.

Does that help explain that portion better? I am still reading through the actual paper, but the SDQ stuff made sense to me at first.

[KLSymph]

Well, I found one study that tends to agree with the paper about this "SDQ" as a useful indicator of "something" in children. [...] That gives them the ability to say the SDQ has High Validity and reliability, since at least one other article claimed something similar.

"It's right because other people do it" isn't a great way to substantiate the usage of the Strength and Difficulties Questionnaire, or to substantiate things in general.  While using it may or may not be right, if it's not right then lots of other people using it makes it seem more right than it is.  That's not a very scientific way to do a study.

Now, as far as I can tell, the SDQ is measuring only one thing. The delta, or change, between a five year old's social and emotional state (as measured by the parent's opinion)  between age 5 and 7.

Well, the paper is using the SDQ to measure the delta.  The SDQ itself does not measure delta, it only measures a snapshot at the time of filling out the questionnaire.  Computing a delta based on this is suspect.

For everyone following along, you should open up the SDQ UK-version questionnaire and scoring instructions, available for free here.  I'll be referring to the "One-sided SDQ for parents or teachers of 4-17 olds" and "Scoring instructions for versions completed by parents or teachers" PDF documents, as I assume this is what the paper uses.

The SDQ is scored as 0, 1, and 2 on its behavior subcategories, it doesn't measure those categories based on a number but on observation of "not true", "somewhat true", and "certainly true" by a parent.  This is, first of all, subjective, because different parents will draw different lines between the responses, but at least the effect of subjectivity may be diminished by having large sample sizes, as the paper uses. The bigger problem is that the SDQ scores the three responses into 0, 1, and 2, but this is an ordinal measure.

Ordinal measures are basically rankings; ordinal datapoints take the operations of "is one greater than the other?", but they can't take the operation of being added together.  The result of addition (and thus subtraction) is not necessarily meaningful.  The SDQ combines five subcategory scores of 0-2 to create a composite score range of 0-10.  As you point out, a zero combined score in the hyperactivity questions probably means low hyperactivity, while a 10 means high hyperactivity, but what is a 5?  Is it a "somewhat hyperactive in all criteria"?  Is it a "certainly hyperactive in half the criteria, not at all hyperactive in the others", and if so which ones? This is impossible to tell from the combined score even though those conditions are not similar.

Even if we accept on faith that the SDQ's numerical scoring is interval measure instead of ordinal, which would allow the additions for the total scoring, we can't meaningfully subtract two total scores to produce a delta, because this kind of delta declares that the zero value has meaning, which makes the delta a ratio measure, which is even worse.  According to the scoring guide, the Hyperactivity Score can be interpreted as 0-5 normal, 6 borderline, 7-10 abnormal.  Let's look at what a delta of +2 would mean based on this interpretation.  If child A goes from 0 to 2, he'd be "still very normal".  If Child B goes from 4 to 6, he'd be "became borderline".  If Child C goes from 5 to 7, he'd be "became abnormal".  By the official interpretation, a delta of +2 can mean a child "didn't change much", "became a little worse", or "became a lot worse".  So can +5, by the same argument, or -5.  In the entire delta range of -10 to +10, everything from -5 to -2, +2 to +5 is ambiguous by a strict reading of the official SDQ interpretation guide.  That's 8/20 delta scores, or 40% of the scale.

What conclusions can you derive from datapoints that can span so far in interpretation?  I'd take a look at the delta scores the paper reported to see if this kind of ambiguity exists in the questionnaire results, but the paper doesn't report them anyway (Table 3 is a year 7 snapshot, but the same metrics for year 5 are not given, and the original data is not provided).

It's bad statistics.

[Ergoemos]

"It's right because other people do it" isn't a great way to substantiate the usage of the Strength and Difficulties Questionnaire, or to substantiate things in general.  While using it may or may not be right, if it's not right then lots of other people using it makes it seem more right than it is.  That's not a very scientific way to do a study.

My apologies for not being clearer. I was trying to be sarcastic with that remark, that they were making claims without states basis in fact.