I favour the "let the author decide" (whether to attach datasets before or after publication) option. The delay from submission to publication is based on low N and could increase. Besides, due to probability theory, some papers will inevitably sit in the forum for a much longer time than others, increasing risk of priority conflicts.
An alternative is that the reviewers can request that the datafile be sent via email, thus not forcing the author to publish the dataset before publication.

The author emails the dataset to reviewers before publication and posts it in the forum after publication. Readers do not need to see the file before the paper is published, only reviewers do.

Then even non appointed reviewers can request the file to be sent via email. At least the author wary of conflict of interests will have proof (email).

In the last few months I have requested data from psychology researchers and I have been ignored, denied, or asked to pay a large sum of money. While annoying, my thought is generally if the data is theirs they can do with it as they choose. However, in each of these cases the individuals teach at public institutions and the data was collected via a federal grant. It isn't just that the researchers are not abiding by the scientific ideal of openness; they are skirting the directives of the grant to share data.

The stand for openness is the primary reason I am excited and optimistic about this journal. However, if the stand is too stringent, it can stifle the success of the journal. Many important data files are open to the public, but can't be redistributed. If the use of these data files is not allowed by the journal because of policies of data sharing, the number and quality of articles that can be published here will be negatively affected.

It seems to me if the data is public, but non-distributable supplying a link to the data should meet the data sharing criteria of the journal. Note this doesn’t include data that is “public” in the sense that you need to be from a member institution or fill out an application to access the data.

Then even non appointed reviewers can request the file to be sent via email. At least the author wary of conflict of interests will have proof (email).

That's a good option. Also emil, I don't think you need to be too stringent with the data sharing. The author can also publish the covariance matrix (+ variance, Mean, SD). With that, the reviewers can use them and replicate the results. Even this is really an improvement. If you look at most studies in other journals, you don't see the covariance matrix reported. Apparently it's not obligatory. But it's a defect. Several times, in meta-analyses, the author(s) mention in their inclusion criteria that they select data for which they have the variance/covariance matrix.

By the way, I remember 2 times when I asked data (by mail) to some authors. I knew they wouldn't give me the raw data, so I asked only for the correlation (or covariance, or both) matrix. I have been told by them that the data do not belong to them and that even this is not possible. Once, I have been told that they don't have the data.

Stats error is unlikely to go undetected by reviewers if the "input data" (e.g., covariance matrix) is reported. The only little disadvantage in it is if the author itself misreported the numbers that appear on the input data.

If you do PCA, path analysis, or multivariate genetic analyses by way of DF regressions, for example, just having the covariance (or the cross-twin covariance for genetic analyses) is sufficient to ensure that you can replicate the same numbers that appear in the submitted paper.

I don't think John is planning to do a factor analysis, and I don't see the option for doing it, in the NCES database we use. We only make tabulations. Thus, the only thing we can do is to add a supplementary paper in which we illustrate the procedure, with screenshots, like the ones I sent you before. We can also put some screenshots of the numbers generated by the tabulations, as an example, but I don't think we will insert all of them. There are too many tables.