The paper creates a battery of items involving conspiracy theories and factor analyses them to find the underlying dimensions of conspiracy theoretic beliefs. The author shows a pattern of dependence to these underlying dimensions, for example finding individuals that are high in flat earth theory are also high in general conspiracy scores, but the reverse ordering is not true. The authors suggest this may imply information about the epistemological journey that people take towards conspiracy theoryies. I greatly enjoyed the paper. There are many details of interest, such as the negative relationship between belief in the validity of IQ tests and the propensity to believe Jewish conspiracies.

There are some areas where I thought the reasoning to be unclear or potentially faulty. Moreover, there were some writing errors that should be ironed out before publication (e.g. notes left in from when drafting the manuscript). As such, I recommend revising the manuscript before it can be accepted.

Below I’ve written a series of comments I had whilst going through the paper. 

It is also unknown whether different varieties of conspiracist belief are predicted by different psychological variables. For instance, groups that vary widely in educational attainment may nonetheless believe the same conspiracy theory at similar rates, as demonstrated in King et al. 2021.

The second sentence does not follow the first here. That King et al. (2021) found a u-shaped relationship between education and vaccine hesitancy does not seem too related to the fact that we don’t know if beliefs in different types of conspiracy theory are predicted by different psychological traits. Maybe make clear what the logic is here - ie. conspiracy theories may not have a simple, monotnic relationship with IQ/SES as one might expect?

The survey is flawed, however, as respondents may have equally conspiracist beliefs regarding extraterrestrials and yet get coded in opposite ways. For instance, some conspiracy theorists believe that the existence of aliens is being covered up, while others believe that aliens are a hoax as part of a psychological operation. Compared to the mainstream position – that governments are telling the truth about the nonexistence of aliens – both positions are equally conspiratorial, and yet only those who believe in the existence of aliens would score highly on this factor, which makes up 20% of the score on this assessment. Consequently, the Extraterrestrial Cover-Up factor correlates least with the overall scale score, at 0.73, while the other four factors correlate at a minimum of 0.87 (average 0.90). 

I think flawed is a strong word and I don’t think it is a major issue. The explanation of your critique is a little unclear. I think you are arguing that having multiple items measuring contradictory conspiracy beliefs undermines the validity of the scale. The reasoning of this is not too clear to me.

Thus far no studies have made an effort to assemble a large variety of questions representative of the diverse range of conspiracist beliefs that are endorsed by self-identified conspiracy theorists. Mutually contradictory questions are included, such as that JFK was assassinated by multiple shooters or that his death was faked, to reflect that fact that multiple mutually exclusive conspiracy theories may exist about the same event. 

I don’t think this is really a problem, conspiracists will respond yes to one of the two items and non-believers will respond no to both items. Either way adding up lots of items like these should get you a good measure of someone’s level of belief in conspiracy theories. In fact, including such contradictory items might be useful to capturing the various dimensions of conspiracy belief.

Previous research has also treated conspiracist belief as a monological construct due to the high correlation between individual questions on surveys. While it is true that there is a central dimension of conspiracist belief, the present study indicates that conspiracist belief is comprised of distinct constellations of beliefs, likely with different epistemological and psychological foundations. 

Monological is the wrong word here. See the definition here: https://www.oed.com/dictionary/monological_adj?tl=true

I think saying previous research has viewed conspiracist belief as unidimensional, or fail tp discuss  it’s factor structure might be more accurate. Besides, I don’t think your criticism is correct in light of Brotherton’s study which performs exploratory factor analysis on conspiracist beliefs and identifies multiple factors, as you mention.

A total of 191 respondents completed the survey. 15 surveys were discarded due to the respondent failing the survey validation question

What’s the validation question?

The age range was 15 to 100, with a mean of 42.9 and a standard deviation of 16. 

It might be useful to see the summary statistics visually or in a plot. An age of 100 sounds like an error or someone giving troll answers. It might good to know this was unusual. And the rest of the respondents were under 80 perhaps.

It is not possible to say what percent of the total variance is accounted for by the six-factor model due to the correlated nature of the extracted factors, however the first six principal components collectively explained 65.61% of the total variance. 

I think this is incorrect. It is possible to report the variance explained by multiple, correlated latent factors. 

In Table 2 it looks as though the below items were used in computing the latent variables. This seems to contradict the Methods section. If the items were not used to compute the factors, I think their correlations with the latent factors should be reported separately.

62. IQ tests are not a valid measurement of intelligence.

38. Jesus was a real historical person

I think you should probably label the axes in Figure 3. I’m not familiar with the method, but one axis will capture more variation than the other, so that should be made clear

Comments on writing errors

Although many psychological constructs have been assessed for their correlation with conspiracist belief, no research has specifically assessed the correlation between trust in elites and authorities and conspiracism.

I find this claim unlikely. From a very quick search of Google Scholar I can find a study relating trust in political elites to conspiracy beliefs.

https://onlinelibrary.wiley.com/doi/full/10.1111/spsr.12270

“the unnecessary assumption of conspiracy when other explanations are more probable”(Aaronovitch,...

Missing a space between the quote and citation

as demonstrated in King et al. 2021.

Missing parentheses around King et al.

 (Yang et al., 2021))

Yang and all has an extra parenthesis

Previous work in understanding the psychometric structure of conspiracy belief has relied on survey questions that are either worded in a general, non-specific fashion, or else focus on a small number (possibly singular) of events, such as those of 9/11 in the US or 7/7 in the UK [insert studies here].

Indeed studies should be inserted here

The first 15 questions on the survey were Attitude questions that assessed the respondents attitudes towards institutions. 

The next 85 questions were Belief questions, assessing the respondent’s agreement with objective statements, e.g. “Humans have never landed on the moon.”
 

Debatable whether to capitalise Attitude and Belief. It’s probably fine.

>. As the name suggests, this survey uses questions that are general and non-event-based, and finds five highly correlated factors, named Government Malfeasance, Extraterrestrial Cover-Up, Malevolent Global Conspiracies, Personal Wellbeing, and Control of Information. The survey is flawed, however, as respondents may have equally conspiracist beliefs regarding extraterrestrials and yet get coded in opposite ways. For instance, some conspiracy theorists believe that the existence of aliens is being covered up, while others believe that aliens are a hoax as part of a psychological operation.

Funny enough, I did some analysis of that data and came to the same factor structure the authors found, though your criticism of the survey itself still rings true.

>. Box plots for attitudes and beliefs scale scores broken down by identity as well as histograms are included in figure 1.

Can you make it more clear what you're measuring here? Standardized differences between non conspiracy theorists and conspiracy theorists would also be nice to see.

Table 1 should also go in the materials/methods.

>Table 2

I think that an exploratory bifactor analysis would be appropriate here. There are psychological/social reasons why some people who endorse one conspiracy theory are more likely to endorse another one, and some beliefs may cluster with other ones due to shared origins/communities/logical reasoning.

Another thing you should highlight in your paper is that some theories load positively onto some conspiracies but not others. E.g. believing JFK died to multiple shooters loads positively onto factor 2, but negatively onto factor 5. 

Besides this, I think most of my other criticisms were shared with prior reviewers.

I am working on updating the manuscript. Some preliminary comments based on the reviews thus far:

Reviewer 1:

1. The data is available in an OSF bucket here: https://osf.io/cn9ds/. I had to add this as a public note when submitting the report as I do not recall seeing a link for data and the data was in a separate bucket from the supplemental materials .I will be releasing the code as well with the next revision.

2. No money was spent on this study. The intent behind this study was to sample from self-identified conspiracy theorists, which are already a small percentage of the population. Additionally, as some conspiracy beliefs are more marginal than others, it would be very difficult to sample from them using a representative sample. If 1/1000 people are Flat Earthers, it would be very challenging to acquire a sufficient sample size form a representative Prolific sample. Conspiracy theorists are inherently a difficult group to study as they tend to be suspicious of all attempts at data collection.

3. I agree that Pearson correlation is suboptimal, I will update the measure.

4. Median time to complete was 14 minutes and maximum time was 138 minutes. I will add this to the report.

5. FUF is First Unrotated Factor in the factor analysis.

6 & 7. I will review this. Oblimin was chosen since the theory called for an oblique rotation. There is a great deal of researcher degrees of freedom in constructing factor analysis, I would consider this one such degree of freedom.

8. Figure 3 presents a dimensionality reduction of the 100-dimensional space of survey responses with colouring representing the score that individual received in each factor. The axes are essentially meaningless. t-SNE was chosen as it tends to preserve the local structure of the higher dimensional data, and thus preserves clusters. An argument could also be made for using UMAP as a dimensionality reduction technique, although I learned of that technique after already producing the diagrams. PCA was not used as the global structure of the dataset was not considered as important as the local structure (having good clusters is more important than preserving distance between distantly related survey responses).

9. Yes, if you believe the Earth is flat, you likely reject most received wisdom, far more than someone who merely believes that the government stages false flags for political purposes.

10. Terrain theory comes from Bechamp's idea that the composition of the body causes disease rather than germs. It is the most widely held belief in germ theory denialist circles. I will add a reference in the paper. https://en.wikipedia.org/wiki/Germ_theory_denialism?useskin=vector

11. IQ was not explicitly assessed, only educational attainment. However, the demographic questions were optional (conspiracy theorists tend to like privacy) and it is likely the answers to these questions were falsified at a much higher rate than the rest of the questionaire. 

12. Age, sex, educational attainment, religion, ethnicity, and urban/rural location were also included in the demographic questions. I will include the ANOVA results in the next draft of the report. Like I said, these questions are at much higher risk of being falsified, and should be considered less reliable.

Navajowhite Saturn:

Yes I am arguing that conspiracy is not related to IQ or SES in a straitforward way. A low IQ person may think that the COVID vaccine is bad because Bill Gates filled it with nanobots, and a high IQ person may think it is bad because of lack of efficacy, risks of lipid nanoparticles or mRNA tech; both would likely be united in refusing vaccination and being skeptical of the pharmaceutical industry.

The reasoning with the aliens factor is that someone who thinks aliens are a psyop will get a 0 on that factor, despite having decidedly conspiracist beliefs on that topic. Thus using it as 20% of a conspiracy scale is spurious.

I did include contradictory items, in fact the example I gave with there being either a second shooter in the JFK assassination or the assassination itself being entirely faked are taken from my survey. I will clarify that when I update the report.

In the Brotherton survey, with the exception of the extraterrestrials factor, all questions would load onto the Generic Conspiracy factor that I extracted. They miss the Jewish conspiracy, Fakery, and Flat Earth factors entirely.

Monological is a term that is frequently used in the literature I was responding to, see https://www.cambridge.org/core/books/abs/power-politics-and-paranoia/examining-the-monological-nature-of-conspiracy-theories/9EE3EF5214F02E078102411113E9003C or https://www.frontiersin.org/articles/10.3389/fpsyg.2017.00861/full

The validation question was just "Please select "probably false" if you completed this survey truthfully.

I will fix typos and errors with studies in the next draft.

Author has updated the submission to version #2

Author has updated the submission to version #3

I have uploaded an updated version of the report, and the supplemental material at the OSF link has been updated as well (please ignore the double submission, I clicked update twice because it was lagging). I have also uploaded the Python notebook which I used to analyze the data and generate the figures, which can be found at https://osf.io/cn9ds/. The updates that have been made are as follows:

I have described in the Participants and Procedure section how the sampling may affect the results. 

I have explicitly included the phrasing of the validation question in the report.

I have updated my discussion of institutional trust and conspiracy theory to include some citations that were brought to my attention by Reviewer 2. In general I have updated and added multiple citations.

I have changed the reported correlation between the respondent self-identity and attitudes and beliefs scores to Cohen's ds instead. I have also added the actual numbers for means and standard deviations based on identity to complement the box plots. These are all given in new tables.

I have added box plots for the participant ages and time to complete data, and added a summary of the participant ethnicities and religions. 

I have added a paragraph describing the ANOVA that was performed using belief scores and demographic data. No significant associations were found, although ethnicity and religion came closest to significance. The raw ANOVA numbers are included in the supplementary materials.

I have explained FUF (First Unrotated Factor) in the paper before using the acronym so that the tables using it are clearer.

I have reported total variance explained for the model based on the variance explained by the principal components.

I have more clearly explained how the dimensionality reduction embeddings work in the paper. I have also changed the dimensionality reduction technique used from tSNE to UMAP as it is a better technique for the purposes of the paper (it was not used originally because I was not aware of it when first writing the paper). Note that the axes on these embedding plots are still unlabeled because the axes are meaningless, since UMAP is a non-linear technique. The plots are intended to show how the different factors cluster.

I have added a citation and some discussion around what Terrain Theory is (germ theory denial, roughly speaking) and its relationship to conspiracy theory.

I have added a paragraph in the discussion about some questions which load positively on some factors and negatively on others. 

I have also updated multiple paragraph to address the points made by Reviewer 2 and to make the paragraphs in question easier to understand.

I have not altered the factor analysis, as this paper is largely a response to the Brotherton et al. 2013 paper, and I have attempted to match their analysis. As such, I believe the method of factor analysis chosen is justified, however I welcome anyone else to analyze the data set in a different way to confirm or challenge the results I have obtained.

Pleaes also forgive the slightly wonky vertical spacing around the figures and tables - I did not originally format the paper in a way that made it easy to fix such issues.

I think the new paper looks good. My only request is that you label the x axis and y axis of the UMAP figure.

Thanks. I will add the labels "UMAP Dimension 1" and "UMAP Dimension 2" to the UMAP figures. I will update the submission once the other reviewers have had a chance to comment on the current version.

There is still a missing parentheses around the year 2021 for the citation King et al. (2021). The author hasn't gone into much depth to describe how the article has been changed in response to my comments.

I encourage the author to review the comments for areas that haven't been cleared up (e.g. the citation typo), then to give slightly more description regarding how the comments have been implemented or why they have not been implemented. From the prior replies and the existence of typos that were already pointed out, it is unclear whether the comments have been taken seriously. Even when the reviewer is wrong or plain stupid, it can be helpful to make changes to prevent other readers misinterpreting parts of the study. Sadly also reviewers are only human and a bit of flattery, with thanks or clearly showing the comments were seriously considered, can go a long way. 

After such a reply, I will be happy to accept the submission.

There is still a missing parentheses around the year 2021 for the citation King et al. (2021). The author hasn't gone into much depth to describe how the article has been changed in response to my comments.

I encourage the author to review the comments for areas that haven't been cleared up (e.g. the citation typo), then to give slightly more description regarding how the comments have been implemented or why they have not been implemented. From the prior replies and the existence of typos that were already pointed out, it is unclear whether the comments have been taken seriously. Even when the reviewer is wrong or plain stupid, it can be helpful to make changes to prevent other readers misinterpreting parts of the study. Sadly also reviewers are only human and a bit of flattery, with thanks or clearly showing the comments were seriously considered, can go a long way. 

After such a reply, I will be happy to accept the submission.

Author has updated the submission to version #4

I apologize for not specifically noting the places where I have implemented or not implemented changes in response to your initial review. This is my first time going through the peer review process so I am learning the etiquette. 

I have made several changes to paragraphs to address shortcomings or a lack of clarity that you helpfully pointed out. I will explain each:

I have clarified that the relationship between IQ and conspiracy belief may not be monotonic in some cases. (I have also added a sentence in the latest version to make this more clear).

I have clarified my opinion on the Brotherton et al. scale, and removed the word 'flawed', as I agree it was likely too strong.

I have clarified that my survey contains mutually contradictory questions that can assess different, mutually exclusive, kinds of conspiracy belief.

I have added a citation to justify the use of the term monological in the context of conspiracy belief. I agree that based on the dictionary definition 'monological' is not the correct term to use, however it is widely used in the psychological literature on conspiracy theory and so I have used it in that sense here.

I have specifically included the phrasing of the validation question in the text of the paper.

I have added summary statistics for age and for time to complete the survey, which are also visually displayed in a box plot, which does indeed show that the age of 100 is an outlier.

I have updated how I report the variance explained by the factor model.

Vis-a-vis the IQ and Jesus questions, they were not used to compute the overall scale score, as in the case of IQ either position could be seen as conspiratorial (e.g. IQ is a conspiracy to promote race differences or something, or IQ research is being suppressed in order to promote egalitarianism), and the Jesus question is not inherently conspiratorial in either case. I included the Jesus question because I have seen it discussed in a conspiracy context before, and to cover as many topics as possible, this being potentially a form of historical revisionism. However, I have included them in the factor scores, as the intent was to see if they fit into any particular constellation of conspiracist beliefs. Based on the data, we can see that the IQ question has an inverse relationship between the Jewish Conspiracy factor and the other factors. It seems that the Jesus question did not strongly relate to any constellation of conspiracy beliefs and loads weakly on the climate change factor, which is the least cohesive factor.

I have now added labels to the axes of the UMAP embeddings.

You correctly noted that I was wrong in stating that no research has yet assessed the relationship between conspiracy belief and trust in elites. I have amended that statement to cite some relevant studies, including the one that you linked, while noting that the measures those studies used did not assess trust in charities or the scientific establishment.

I have fixed the missing space between the end of the quote and the Aaronovitch citation in the introduction.

In this latest version I have added parentheses around the year in King et al. (2021). I agree this was a mistake in the styling.

Regarding the Yang et al., 2021 citation, there were two parentheses because the citation was nested in an existing parenthetical; I have reworded the sentence to avoid the awkward grammatical construction.

I have inserted citations to the studies that were focusing on 9/11 and 7/7 where previously I had left a placeholder. Thank you for noticing this.

I have not updated the capitalization of Attitude and Belief, which you agreed was probably fine.

I appreciate your initial review, which spurred many revisions and corrections in the updated versions of the paper. Please let me know if you have any lingering concerns.

Thank you to all the reviewers for taking the time to read the paper and leave helpful comments, and thank you for accepting it!

My understanding was that the paper was accepted and is awaiting typesetting. If I can help anything by providing the raw HTML or figures I will happily do so, otherwise I'm not sure what other updates would be required.

The submission was accepted for publication.