Progress has been made, but, sorry, but I am still not satisfied.
while military intervention in the Middle East has a fairly strong relationship with the first (d = 0.69–1.63, r = .40; β = 0.39–0.45) and second measure (d = 0.53–1.46, r = .41; β = 0.28– 0.46), but a less consistent relationship with the third (d = 0.39–2.37, r = .30; β = 0.33–0.52).
Both hypotheses received some support from the analyses. Percentage of Muslims in the population (logged) had a strong association with the first and third measures of terrorist threat, but a somewhat weaker association with the second. Military intervention in the Middle East has a fairly strong relationship with the first and second measure, but a less consistent relationship with the third: although all three measures of military intervention were significant in the multiple regression models, only part of anti-ISIS military coalition had significant bivariate association with log of 1 + number of arrests for religious terrorism.
What the seems looks to be talking about is the consistency of the effect. But as can be seen, the effect is 100% directionally and size-wise fairly consistent. What is not consistent is whether the p-value is below alpha or not, but of what interest is this? Given the effect size and the sample size,
even if there is a true effect, we would not have a p-value < alpha in all cases.Let me explain why I think it is a problem calling this a lack of consistency by quoting the classic Schmidt and Hunter (1984) paper:
http://onlinelibrary.wiley.com/doi/10.1111/j.1744-6570.1984.tb01453.x/abstractBy the end of World War 11, hundreds of validation studies had been conducted to determine the extent to which ability tests predict performance on different jobs. When these studies were compiled by Ghiselli (1966),he found marked variation in validity coefficients even across sets of studies in which very similar tests were used to predict performance on seemingly similarjobs. Ghiselli concluded that most of this variance was apparently due to specific factors that determined the nature of job performance in particular settings. This situational specificity hypothesis of employment test validities predated Ghiselli’s research; the effect of Ghiselli’s findings was to provide seemingly strong support for that hypothesis.
The situational specificity hypothesis, as it has existed historically, makes two predictions, one well known and obvious and one that is less well known and more subtle. The well known and obvious prediction is that there will be substantial variation in validities across settings and organizations for the same and similar tests and jobs and this variation will be real. Research in validity generalization has challenged this prediction by demonstrating that most, and in many cases all, of the variance in validity coefficients across studies is due to statistical and measurement artifacts, principally sampling error. The alternative hypothesis tested has been that situational specificity of job performance is either nonexistent or has little or no effect on validity coefficients of ability tests used in employment, and that, as a consequence, employment test validities are generalizable. In recent years, research evidence has accumulated indicating that the validities of numerous employment selection procedures are indeed generalizable across settings, organizations. and time periods. For summaries of the research on validity generalization, see Schmidt and Hunter (1981), Hunter and Schmidt ( 1983), and Linn and Dunbar (1982)
In other words, using language like this ("less consistent") invites the error that plagued the earlier studies of validity of tests for personnel selection: attributing sampling error to moderator effects whereas none existed.
So, I would be happy if the author either: 1) clarified that it was only the p-value that not always < alpha, as expected by sampling error, OR 2) removed the mentioning of the lack of consistency of the p-values.
---
I have yet to re-do my analytic replication because I am swamped with work from another project. I will get it done.