Journal of Political Economy Microeconomics

Scientific research aims to create a stock of knowledge. Optimally adding to this stock requires confidence in the received empirical estimates. Increasingly, social scientists are generating their own data. Lab and field experiments have taken their rightful place in the empirical spectrum, and more recently there has been a growing line of research using data generated via surveys. What is often missing from such studies is the necessary detail for the reader to determine key information about data generation, for example, i) the target population sampled, ii) the way people are sampled and recruited (and if there is any replacement for non-response), iii) non-response rates (overall and by covariates), and iv) the researcher or survey firm’s method to deal with non-response (or refusal to answer some question)?

To address gaps in information about data generation, the editors of JPE Micro have developed a set of guidelines that will be put in place in early 2024. Authors of studies that generate their own data will be expected to register their project before data collection in one or more of the various trial registries and write a pre-analysis plan (PAP) before data collection. If a study is not pre-registered and has no PAP, then it should not be submitted to JPE Micro . Studies submitted without the required pre-registration and PAP will be sent back to the author without a review. ²

Pre-registration is a publicly documented statement summarizing the study and the hypotheses the researcher plans to investigate before data are collected and analyzed. A subset of pre-registrations are PAPs , which offer greater detail about the researcher’s plans. ³ If used appropriately, registries and PAPs can help to tackle key issues in the credibility crisis such as p-hacking and the file drawer problem.

Pre-Registration
The following Registration Checklist should be used in creating your pre-registration. More information may be included at the author’s discretion:

• Experiment Name
• Country, State, or City
• Short description (< 280 characters)
• Intervention Start Date
• Intervention End Date
• Outcomes (Surrogates and End Points)
• Experimental Design, including treatment arms
• Planned Number of Observations
• Sampled Population
• Was IRB approval obtained (or exempt)? If approved:
  • IRB Name
  • IRB Approval Date
  • IRB Approval Number
• If exempt from IRB approval, please provide a brief explanation.

The AEA Registry is our preferred registry. However, social scientists have created a variety of specialized trial registries, summarized in Exhibit 1.

Exhibit 1: Various Trial Registries

Pre-Analysis Plan
Another pre-intervention tool with a potential impact on knowledge creation is a pre-analysis plan (PAP): a subset of pre-registrations that offers greater detail about the researcher’s plans to analyze data before the intervention starts. Note that although they mean the same thing in some disciplines, in economics a registration is typically distinct from a PAP. While registration provides metadata and a list of primary outcomes and treatment arms, in contrast, a PAP goes beyond registration by specifying a set of primary analyses and the content of those analyses. Oftentimes, PAPs are filed with trial registrations, as registries can provide a credible timestamp.

An effective PAP will answer two detailed questions: “What are the key outcomes and analyses?” and “What is the planned estimator and statistical test for those outcomes?” A more detailed pre-analysis plan should go even further and specify all the steps involved in analyzing the data. A useful 9-item PAP checklist includes:

1. Well-defined research question and hypotheses
   Clearly articulate your research question(s) and the hypotheses you plan to test. This is not only good for transparency but augments good research practice. A problem well-conceived is half-solved.
2. Relevant insights on how you will shed light on received data
   Clearly describe your sample selection. Summarize the population you plan to draw data from, relevant inclusion/exclusion criteria, and any type of blocking you plan in your randomization. In addition, please discuss the key mediation paths and moderators that will be explored, and provide details where possible, e.g., “question Q1 provides relevant moderators, and questions Q2 and Q3 provide the two tests of mediation paths that will be explored.”
3. Details on key design decisions to maintain experimental integrity ⁴
   Summarize balance tests and how data will be cleaned, including information on how missing data will be handled, how outliers will be treated, and any transformations or aggregations.
4. Summarize all data sources and variable construction
   Describe the data you plan to generate in terms of specific outcomes, covariates, sample size, and collection methods, and relevant details about other data sources you intend to analyze. This provides scientific integrity to your approach and includes describing the process of treatment assignment, randomization approach, and relevant power tests. In addition, please specify how variables will be constructed and examined (e.g., a linear specification versus log-linear).
5. Statistical Approach: Outcomes
   Clearly define the primary outcome(s) on which you will focus. Define how you will measure these outcomes, including any operational definitions or indices. Describe how you will estimate treatment effects and report key statistics, such as p-values, effect sizes, and confidence intervals. If you plan to examine secondary outcomes, please specify them and explain their importance (whether as actual outcomes or mediators).
6. Statistical Approach: Methods
   Explain the statistical methods and models you plan to use, and include details on key estimation issues such as type of estimator and implementation details. If any machine learning or nonparametric procedures will be used, please explain how training/test splits will be obtained, how tuning/hyperparameter selection will be performed, and provide other specific algorithm-based information. Specify any control variables, covariates, or other adjustments to be made, including aspects such as MHT corrections or attrition corrections. (For some types of studies, it should be possible to write the analysis code in advance.) Include equations, if applicable, and specify the approach to estimation – for instance, a difference in means approach or a difference-in-differences approach – and if you plan on exploring other experimental estimands.
7. Statistical Approach: Sequencing
   Describe the sequence of analyses you will conduct, including relevant theory testing, mediation path analysis, and any subgroup analyses or robustness checks. Specify the order in which tests will be performed to minimize data-driven decisions.
8. Ethics
   Discuss ethical considerations relevant to your research, such as informed consent, data governance and management, and any Beneficence, Respect for Persons, and Justice issues.
9. Timely Registration and Milestones
   Include your PAP in your pre-registration, and archive it appropriately in the AEA registry or one of the other registries noted above. In doing so, provide a timeline for your research project, including milestones for data collection, analysis, and reporting. In addition, please include your PAP with your JPE Micro submission .

Authors may generate insights that enter the realm of exploratory (or non-pre-registered and PAP) results, and we welcome discussion of those results in the study. We ask that inclusion of such results are qualified with a statement such as: “these results are beyond our PAP.” We believe that following the PAP should be the ideal and the default, with deviations from it requiring justification.

Registered Reports 
The JPE Micro offers a Registered Reports (RR) track where authors can submit their prospective empirical projects and have them reviewed and in-principle approved for publication before the data is collected and the results are known. Registered reports go a step further than pre-registration and PAPs in that they are designed to tie both the researcher’s and the journal’s hands by having the researcher submit a detailed plan to a journal before any analysis – and this needs to be credibly shown. We believe that this is one of the solutions to publication bias in science.

JPE Micro follows the common two-stage procedure for RRs. In Stage 1, authors submit a proposal, which is then peer-reviewed. If the outcome is positive, then the paper is in-principle accepted. After data collection, analysis, and final write-up, the manuscript goes through another round of peer review in Stage 2, with a focus on whether the PAP has been followed and whether the integrity of the experiment was as promised. Finally, following possible revisions and editor approval, it is published.

We envision RR manuscripts to resemble informative PAPs and to, in the least, follow the 9-item PAP checklist detailed above. Going beyond a PAP, we encourage authors to discuss relevant literature, explore where their research places in the literature, and share potential shortcomings. Non-pre-registered results potentially can be added in Stage 2, where those results are exploratory and not confirmatory. In many ways, the RR Stage 1 manuscript will resemble an ideal “normal” manuscript without a results section and a clear conclusion.

For lab and field experiments, we expect and prefer RR manuscripts to be submitted prior to any data collection so that reviewers have a chance to comment on the design, variables, and analysis. If data collection has already started (but the authors do not have access to the data), the manuscript can still be sent out for review but reviewers (and editors) might raise concerns with measures, etc. For other types of data (e.g., a study using a quasi-experimental design with administrative data), we expect RR manuscripts to credibly describe that the authors have not had access to the data.

Allowing RR submissions places a particular trust in authors not to use the editors and referees of JPE Micro for their comments and then shop the finished product to other journals. In this spirit, our policy is strict: authors of accepted RRs can submit their study to the Journal of Political Economy as a de novo submission or they may send it back to JPE Micro. Submission to any other journal is strongly discouraged. We reserve the right to create a public list of authors who violate this policy.

Observational Data
Many empirical studies do not generate their own data but use secondary data as their primary source. While the timing of such empirical endeavors renders them difficult to place into our framework above, we urge authors, when possible, to follow the above recommendations. For example, when using administrative databases, there might be a moment before gaining access to the data wherein the author can create a pre-registration, PAP, and RR. This example comes from a general class of data where there is a timing element, akin to the experimental approach, that permits the researcher to document their plan before having access to the data. We strongly urge authors to do so.

¹ These guidelines follow List (2024), Experimental Economics: Theory and Practice , Univ. of Chicago Press.

² We do allow certain exceptions in cases where an organization conducted the experiment before the researcher/organization partnership commenced, where organizational privacy was at risk, and instances where other reasonable constraints caused non-compliance. In such instances, we ask the authors to include such information in the opening footnote of the study and in a cover letter that accompanies the submission.

³ This is only the case in economics – in, for example, psychology, pre-registration and having a pre-analysis plan means the same thing. While their separation has caused some confusion in economics, we wish to emphasize that both a pre-registration and PAP are expected.

⁴ For example, satisfying the four exclusion restrictions, SUTVA, Observability, Compliance, and Statistical Independence (see John List, chap. 3, in Experimental Economics: Theory and Practice , Univ. of Chicago Press, 2024).