ACADEMIC WRITING GUIDE
How to Assess Publication Bias and Interpret Funnel Plots in a Dissertation (With Worked Examples)
A meta-analysis can be carefully calculated and still give a misleading impression if the evidence available to analyse is an unrepresentative subset of what researchers actually studied. Small trials with encouraging results may be easier to find than equally relevant trials with disappointing or unclear findings. Sometimes the study is published but a particular measured outcome is missing from the report. These are problems of missing evidence, not merely of study quality.
A funnel plot can be a useful clue. But it is not a lie detector for publication bias: asymmetry can also reflect genuine differences between small and large studies, study methods, chance or statistical artefacts. This guide explains how a dissertation author can investigate potential publication bias, decide whether a funnel plot is informative, document missing results and report a proportionate conclusion.
The guidance draws primarily on Cochrane Handbook Chapter 13, the BMJ recommendations for interpreting funnel plots, and the PRISMA 2020 statement.
Teaching-example disclaimer: All studies, records, comparisons and numerical values below are fictional. The ten plotted points are constructed demonstration data, not extracted results, and the plot is a schematic teaching display rather than statistical output. The separately illustrated event-count calculation is likewise fictional. No pooled meta-analysis or formal funnel-plot asymmetry test was fitted. The examples demonstrate arithmetic and reasoning, not an intervention's actual effects.
Publication bias, reporting bias and missing evidence: what differs?
Publication bias is selective availability of whole studies related to their results. A trial with striking findings might appear in a journal while another completed trial with less attractive results remains unpublished. The reviewers then observe a potentially biased collection of studies.
Selective non-reporting of results arises when a study is available but one or more outcomes, time points, subgroups or analyses are not fully reported, potentially because of the results. For example, an article might describe satisfaction outcomes but omit a previously specified academic-performance outcome. An absent number is not proof that the authors suppressed it, but it is a reason to examine protocols, registries and supplementary materials.
Small-study effects mean that estimates from smaller studies differ systematically from those from larger studies. This pattern can arise from publication bias, but also from differences in participants, interventions, study quality, design or simple chance. A funnel plot is principally a display that may suggest small-study effects; it does not directly identify the mechanism.
Risk of bias due to missing evidence is a judgment about whether unavailable studies or results could distort a particular synthesis result. This synthesis-level question is different from the risk-of-bias judgment made for an individual included study. See Acadelyra's guide to assessing risk of bias in a systematic review for that separate level of appraisal.
| Concept | What might be missing or different? | What to examine | What not to conclude automatically |
|---|---|---|---|
| Publication bias | Entire completed studies | Registries, protocols, dissertations, reports, search coverage | That every unpublished study had an unfavorable result |
| Selective result non-reporting | A prespecified outcome, time point or analysis | Protocol versus final report, supplements, author correspondence | That every incomplete report deliberately hid a result |
| Small-study effects | Effect estimates differ by study size or precision | Plot shape, contextual differences, design and heterogeneity | That asymmetry proves publication bias |
| Missing-evidence bias in the synthesis | Available evidence may systematically differ from unavailable evidence | All lines of evidence and plausible consequences for the result | That a single statistical test settles the judgment |
Before making a funnel plot, investigate the evidence you could be missing
A funnel plot should not be the first or only method used to assess reporting biases. Begin with the review question and the search strategy. Were trial registries and relevant grey-literature sources consulted where appropriate? Were publication dates, language restrictions and database choices justified? Do the reference lists of included studies reveal relevant unpublished reports? Are there protocols or registry entries for studies that appear completed but have no corresponding results?
For each potentially eligible record, distinguish an unpublished completed study from a study that is still ongoing, a duplicate report, and a study that fails your eligibility criteria. A record of an ongoing study is not an unreported negative result. Similarly, one study may have multiple publications and should not become multiple independent studies in a meta-analysis.
Next, compare prespecified and reported outcomes where documents are available. If the original protocol listed a relevant outcome that is absent in the paper, record what is missing and whether the review team attempted to retrieve it. An unavailable effect estimate may simply reflect a reporting limitation; any inference about why it is missing should remain conditional.
Acadelyra's guide to handling missing study data explains how to log those gaps and avoid inventing missing results. A robust data extraction form can also keep outcome availability and correspondence traceable.
Build an evidence-availability log
| Fictional record | Observed evidence | Appropriate next step | Permissible interpretation |
|---|---|---|---|
| Trial Cedar | Registry lists a completed eligible trial; no results found | Search reports and contact investigators | Potentially missing study; direction of its result unknown |
| Trial Willow | Final report lacks a protocol-listed primary outcome | Check supplement and request outcome data | Possible selective non-reporting; reason not established |
| Trial Pine | Abstract and journal paper describe the same participants | Link as multiple reports of one study | Do not count twice |
| Trial Elm | Registry says recruitment is ongoing | Record ongoing status and eligibility | Not evidence of a suppressed completed result |
| Trial Ash | Published result available for review outcome | Extract as specified | An available result, not proof that the body of evidence is complete |
A real dissertation should record identifiers, search dates, search terms, eligibility decisions, correspondence dates and results. A simple audit log prevents a missing report from being quietly converted into a fabricated effect estimate.
What does a funnel plot actually display?
A conventional funnel plot places each study's effect estimate on the horizontal axis and a measure of its precision or size on the vertical axis, often the standard error. More precise estimates tend to cluster toward the top, while smaller, less precise studies often spread out more widely toward the bottom. If effects come from a common underlying distribution without systematic size-related differences, the cloud of points may look roughly like an inverted funnel.
That is a heuristic shape, not a formal diagnostic guarantee. A visually asymmetric plot may signal that smaller studies have systematically different estimates, but it cannot say whether the reason is selective publication, clinical context, higher risk of bias, different interventions or random variation. A symmetric plot also cannot establish that no results are missing.
The horizontal scale matters. Ratios such as odds ratios typically need a suitable transformed scale for statistical analysis, and effect direction should be consistent across studies. A plot that combines incompatible outcome measures can produce misleading patterns before publication bias is even considered. The Cochrane meta-analysis chapter provides the statistical context.
When is funnel-plot interpretation too weak to support a claim?
With relatively few studies, apparent gaps and asymmetry may arise by chance, while formal asymmetry tests often lack useful power. Cochrane and the BMJ recommendations caution against routine funnel-plot asymmetry testing when a meta-analysis includes fewer than about ten studies; even ten is not a guarantee of adequacy. The relevance of any threshold depends on study sizes, heterogeneity, statistical method and effect measure. It is better to state why an assessment is uninformative than to add a decorative funnel plot because a software menu offers one.
With pronounced between-study heterogeneity, the plot can also reflect meaningful differences in settings or designs. Your interpretation should therefore be connected to the forest plot and the review's scientific question, not isolated as a graphic in the Results chapter. See our guide to interpreting heterogeneity.
| Question | If the answer raises concern | Recommended response |
|---|---|---|
| Are there enough comparable studies? | Sparse evidence makes patterns unreliable | State that formal asymmetry assessment is not informative |
| Do studies use compatible effect metrics and directions? | A scale problem can create artefacts | Harmonize valid metrics or avoid a combined plot |
| Are small and large studies clinically similar? | Intervention, setting or follow-up differs | Describe the differences and their plausible consequences |
| Is study quality related to size? | Smaller studies may have different bias patterns | Examine appraisal results without claiming causality |
| Are registered studies or outcomes missing? | Possible reporting bias is supported independently | Document the missing evidence and its plausible impact |
| Is the plot interpreted with model assumptions? | The displayed shape may be misleading | Report analytical choices and limitations |
Worked numerical example: read a funnel plot without diagnosing its cause
Suppose a dissertation synthesizes a binary outcome across ten fictional randomized studies. For a risk ratio (RR), values below 1 favour the intervention if the event is undesirable; the horizontal plotting scale is ordinarily log(RR), not raw RR. For each study, the vertical plotting coordinate below is the standard error (SE) of log(RR). Smaller SE values correspond to more precise estimates, commonly displayed toward the top of an inverted funnel plot.
First, see how an individual study could contribute an effect estimate. In a separate fictional illustration, 24 of 120 intervention participants and 36 of 120 controls experience the outcome. Their observed risks are 0.20 and 0.30, giving RR = 0.20 ÷ 0.30 = 0.667 (rounded). Taking the natural logarithm yields a negative log(RR), which statistical software would calculate from the study counts. An RR below 1 suggests a lower observed event risk in this one hypothetical study; it is not, by itself, proof of an intervention effect or of publication bias. The logarithmic transformation and uncertainty estimates would be calculated in the review's statistical software from the underlying counts.
The following different fictional ten-study dataset is provided specifically to demonstrate a funnel plot. Its log(RR) and SE entries are assumed study-level inputs, not derived from the preceding event-count example.
| Study | log(RR) | SE of log(RR) | Position in plot |
|---|---|---|---|
| A | −0.08 | 0.10 | Most precise |
| B | −0.10 | 0.12 | Near top |
| C | −0.06 | 0.14 | Near top |
| D | −0.04 | 0.16 | Middle |
| E | −0.08 | 0.18 | Middle |
| F | −0.12 | 0.20 | Middle |
| G | −0.16 | 0.22 | Lower |
| H | −0.24 | 0.24 | Lower |
| I | −0.32 | 0.26 | Near bottom |
| J | −0.42 | 0.28 | Least precise |
Here is a schematic text-only view of these coordinates. Horizontal placement is approximate; use the numeric table above when producing an actual scatter plot in statistical software.
log(RR) →
SE ↓ −0.4 −0.3 −0.2 −0.1 0.0
0.10 A
0.12 B
0.14 C
0.16 D
0.18 E
0.20 F
0.22 G
0.24 H
0.26 I
0.28 JStep 1 — describe the observed pattern. The three most precise estimates (A–C) average −0.080 on the log scale: (−0.08 − 0.10 − 0.06) ÷ 3. The three least precise (H–J) average approximately −0.327: (−0.24 − 0.32 − 0.42) ÷ 3. Their descriptive contrast is approximately −0.247 log units (−0.327 − [−0.080], using rounded means). This is not a weighted meta-analysis, effect estimate, Egger test or valid inferential statistic; it merely makes the size-related pattern understandable.
Step 2 — consider alternative explanations. Smaller, less precise studies appear to show a more favourable effect, but the pattern could arise from missing studies, varying participant risk, different intervention implementations, differences in study conduct, heterogeneity or chance. Ten studies meet a frequently used rule of thumb for considering an asymmetry test; that count alone does not validate a test or its interpretation, especially when heterogeneity is present. See the Cochrane guidance on missing evidence and the BMJ funnel-plot recommendations.
Step 3 — decide what the dissertation can claim. A defensible description is: “In this fictional illustration, less precise studies tend to report more favourable estimates. The pattern motivates investigation of small-study effects and missing evidence but does not establish publication bias.” In an actual review, regenerate the plot from extracted study data, use the appropriate axis and effect-scale conventions, check study design and heterogeneity, consider protocols and registries, and report any software-based asymmetry test only when justified.
Worked example: making a careful missing-evidence judgment
Imagine a dissertation review about an online study-support intervention. The student's search identifies a set of eligible published studies; registry searching also finds one completed study with no accessible outcome report. A second published study's protocol lists an academic-performance outcome that is absent from the final article. The accessible studies vary in recruitment setting and follow-up duration, and the smaller studies appear to have more positive reported findings.
Step 1: establish the synthesis question. The student defines the target outcome, follow-up period, intervention comparison and eligible study designs, drawing on the review protocol rather than retrofitting the question to the published results.
Step 2: separate observations from explanations. Observations include the inaccessible completed-study results, the outcome missing from a published report and the apparent difference between small and large studies. Hypotheses include selective reporting, different participant populations, different delivery intensity and chance. Neither observation proves a particular hypothesis.
Step 3: pursue available documents. The student searches the registry, checks protocols and supplementary materials, contacts authors where appropriate, and records responses. No effect value is inserted for an unanswered email. If key data remain unavailable, that limitation stays visible in the extraction log and reporting.
Step 4: decide whether plot-based analysis is informative. Suppose the evidence base is too small and heterogeneous for a reliable funnel-plot assessment. The student explicitly declines to run a formal asymmetry test. That is a defensible methodological decision, not a failure to complete the statistical section.
Step 5: assess possible consequence for the synthesis. If the missing results could reasonably differ from the available results in a direction that materially changes interpretation, risk of missing-evidence bias deserves serious consideration. Yet without the unavailable results, the student cannot quantify the true difference or assert that the original pooled effect is definitely exaggerated.
Step 6: report uncertainty. A responsible conclusion states what could bias the evidence, what was done to investigate it and what cannot be resolved. The review may still describe the available estimates while avoiding claims stronger than the complete evidence warrants.
| Evidence stream | Observation | How it affects judgment |
|---|---|---|
| Study registers | A completed eligible study has no result report | Raises concern about potentially missing whole-study evidence |
| Protocol comparison | A planned outcome is unavailable in one publication | Raises concern about missing study results |
| Study-size pattern | Smaller published studies appear more favorable | A clue with multiple explanations; not proof of publication bias |
| Heterogeneity | Context and follow-up differ | Weakens attribution of any size-related pattern to publication alone |
| Author contact | No usable response received | Missing evidence remains missing; do not impute invented findings |
| Overall synthesis judgment | Important uncertainty remains | Explain concern and implications without claiming a proven mechanism |
Should you use Egger's test, trim-and-fill or another statistical method?
Not automatically. Egger-type regression tests examine certain forms of funnel-plot asymmetry under assumptions that can fail with few studies, heterogeneity, unusual effect measures and dependencies. The name of a test does not validate its application. If your dissertation includes too few suitable studies, or the model is poorly matched to the data, report why you did not use the test.
The trim-and-fill procedure is often introduced as a way to estimate a hypothetically symmetric funnel and an adjusted pooled estimate. That output is not a recovery of the missing real studies, nor is it a definitive correction for publication bias. Because asymmetry has multiple causes, an adjusted estimate can be misleading if interpreted as the true answer. Do not present algorithmically filled points as observed trials.
More sophisticated selection models or sensitivity analyses may be appropriate in specialist work with suitable evidence and statistical expertise, but they also depend on assumptions. For a dissertation, methodological justification and honest uncertainty usually matter more than collecting the greatest number of diagnostic statistics. The BMJ interpretive recommendations and Cochrane guidance are better starting points than an uncritical software checklist.
How to report publication-bias assessment under PRISMA 2020
PRISMA 2020 explicitly asks review authors to describe methods for assessing risk of bias due to missing results (item 14) and present the resulting assessments for each relevant synthesis (item 21). A statement such as “publication bias was absent because the funnel plot was symmetrical” is inadequate. A complete report explains the searched sources, protocol/registry comparisons, data availability, appropriateness of graphical or statistical methods, any findings and what they do—and do not—justify.
Example Methods wording
We evaluated the possibility of bias due to missing evidence for each prespecified synthesis. We compared available study reports with accessible registry or protocol records, documented outcomes without usable results, and recorded attempts to obtain missing information. Funnel-plot assessment was considered only where the number and comparability of studies made its interpretation appropriate. We did not infer publication bias solely from plot shape.
Example Results wording
One hypothetical completed eligible study had no accessible result report, and one published study did not provide usable data for a protocol-listed outcome. We documented the retrieval attempts and could not establish the missing results or their direction. The available evidence was insufficient for an interpretable formal funnel-plot asymmetry assessment. We therefore judged the synthesis potentially vulnerable to missing-evidence bias, with the extent and direction of any distortion unknown.
Example Discussion wording
The review's interpretation was limited by the possibility that some eligible results were unavailable. The documented missing records raise concern, but they do not prove that publication decisions were driven by effect direction or statistical significance. The conclusions should therefore be viewed in light of this unresolved uncertainty rather than interpreted as representing every completed study.
These are illustrative paragraphs, not descriptions of analyses performed for your dissertation. Adapt them to your actual records and decisions, including a named assessment tool when one was genuinely used.
How does this relate to GRADE certainty?
Risk of bias due to missing evidence may matter to the GRADE publication-bias domain, but it is not identical to an individual trial's risk of bias and should not be reduced to one P value. The appropriate certainty judgment depends on the outcome, credibility of missing-evidence concerns and the broader evidence base. The Cochrane GRADE chapter and Acadelyra's guide to GRADE and Summary of Findings tables explain how to keep the assessment outcome-specific and transparent.
Common mistakes that weaken dissertation reporting
Equating asymmetry with publication bias. The shape suggests small-study effects at most; alternative explanations must be considered.
Equating symmetry with an absence of bias. A plot can look balanced even when studies or outcomes are missing, especially with sparse evidence.
Testing every outcome indiscriminately. An uninterpretable test adds a number, not defensible insight. Explain why an assessment could or could not be conducted.
Counting multiple reports as independent trials. One study with an abstract and a paper is not two studies.
Treating unavailable results as zero effects. Unreported effects are unknown; they do not automatically equal no difference.
Asserting motives without evidence. A missing registry outcome can justify concern without proving intentional suppression.
Assuming adjustments reveal the truth. Sensitivity models and trim-and-fill cannot recover unavailable real results without assumptions.
Downgrading certainty mechanically. GRADE decisions require an outcome-level justification based on the relevant evidence, not a fixed rule triggered by a single graph.
Frequently asked questions
How many studies are needed for a funnel plot?
There is no universally sufficient number. A commonly cited caution is to avoid routine asymmetry testing with fewer than about ten studies, but even larger sets may be unsuitable if heterogeneous or poorly measured. Consider the actual evidence and method.
Does a nonsignificant Egger test prove no publication bias?
No. A test can lack power or be unsuitable for the data. A nonsignificant result is not proof of comprehensive evidence or unbiased reporting.
Is publication bias the same as risk of bias within studies?
No. Publication and selective-result reporting can bias the available evidence for a synthesis. Risk of bias within a study concerns whether its reported effect estimate is systematically distorted. Both deserve attention, but the appraisal questions differ.
Should I include unpublished studies?
Include eligible studies and reports according to the protocol and review standards, considering data reliability, duplicate records and ethical or access limitations. Publication status alone is not a defensible reason to exclude otherwise eligible evidence.
What if I cannot locate a protocol or registry record?
Record the absence and any search attempts. Lack of a protocol does not prove selective reporting; it may limit your ability to detect it.
Can a narrative synthesis assess missing evidence?
Yes. The problem of missing studies or outcomes persists even when effects cannot be pooled. Document search coverage, outcome availability, suspected missing evidence and limitations without inventing a funnel plot.
Should I always use trim-and-fill if the funnel plot is asymmetric?
No. It relies on modeling assumptions and must not be presented as verification of missing real studies. Its usefulness depends on the question, statistical method and credibility of those assumptions.
Where should I report the assessment?
Methods should describe the prespecified approach. Results should report outcome/synthesis-level findings. Discussion should explain plausible effects on conclusions. Include appendices containing relevant logs and decisions when useful.
A practical final checklist
Before submitting, confirm that you have distinguished study-level risk of bias from synthesis-level missing-evidence bias; checked protocols and registries where possible; avoided double-counting reports; documented missing outcomes and author contact; evaluated whether funnel-plot methods are genuinely informative; avoided equating asymmetry with publication bias; retained uncertainty where the missing results are unknown; reported the assessment in Methods and Results; and discussed consequences for interpretation and certainty.
The central rule is simple: a visually compelling plot cannot replace a credible account of which evidence may be missing and why that matters.
Final takeaway
Investigating publication bias is not an exercise in proving whether a funnel is symmetrical. It is a structured assessment of the completeness of evidence and the risk that available results systematically misrepresent the full body of studies. A careful dissertation will document missing studies or outcomes, use graphical methods only when defensible, consider competing explanations and state what remains unknown.
For support organizing your review's methods, strengthening transparent reporting or editing the presentation of your own evidence, explore Acadelyra's Research Support services. Researchers remain responsible for their own searches, calculations and academic judgments.
Authoritative references and further reading
Sterne, J. A. C., et al. (2011). Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ, 343, d4002.
Page, M. J., et al. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews.