Combining AI Output Using Ensemble Reasoning

Introduction

There are two main issues with LLM AI output that occur due to the probabilistic nature of LLM models:

• The output is often incomplete.
• LLM responses can contain inaccuracies and inconsistencies.

The question then arises: is it possible to combine multiple AI responses, each of which may possibly be slightly inaccurate, into a single, more accurate and more comprehensive result than any of the individual inputs?

In general, the answer is yes, provided the following conditions are met:

1. Each response has a degree of validity (i.e., is not entirely erroneous).
2. The process of combination is performed with a logical framework that distinguishes overlapping points from contradictory elements.

Expressed mathematically, we can say that by viewing each LLM output as a set of possible solutions or constraints, we can then find their intersection (area of overlap), which defines a more accurate solution space.

• The more overlapping regions we identify among these sets, the more confidence we can have in the convergent subset.
• Contradictions or discrepancies highlight areas requiring further scrutiny or additional external validation.

In other words, when each LLM output is conceptualized as a set of possible solutions or constraints, intersecting these sets narrows the solution space. Overlapping regions indicate a higher likelihood of correctness; discrepancies point to areas needing verification.

Preconditions for Combining AI Outputs

Two main factors are required to integrate multiple imprecise or partially flawed AI responses into a single, more accurate outcome are crucial: the inherent reliability of each individual response and a structured method for merging them.

• Degree of Validity: First, each response should contain at least some elements of correctness or relevance. If a response is almost entirely speculative or riddled with proven inaccuracies, including it in a combined analysis may add more confusion than clarity.
• Structured Logical Framework: Second, the manner in which these responses are compared and fused must follow a well-defined logical framework. The process of combination must distinguish overlapping points (which reinforce confidence in certain details) from contradictory elements (which require further scrutiny or external corroboration).

Together, these two preconditions—response validity and methodical integration—provide a foundation for Ensemble Reasoning. Even if no single LLM output is perfectly accurate, these conditions enable a process through which the combined result can be more robust than any single, partially flawed response would be on its own.

Ensemble Reasoning for Hard Facts and Qualitative Knowledge

Not all LLM outputs deal with the same type of knowledge. Some discuss empirical data (dates, numbers, or verifiable occurrences), whereas others provide subjective evaluations, predictions, or opinions.

Two main approaches are used in Ensemble Reasoning for these two contexts:

Set Theory for Combining Hard Facts

• Constraint Satisfaction: Each factual statement is treated as defining a subset of possible worlds in which that statement holds.
• Set Intersection: Factual consistency is achieved by intersecting these subsets. If all statements can coexist without contradiction, they collectively form a more constrained (and hence more precise) set of potential truths.
• Contradiction Handling: Should certain statements contradict one another (e.g., conflicting numerical values), further checks or external verification is necessary to resolve or discard inconsistent claims.

Abstract Logic for Combining Qualitative Knowledge

• Dialectical Reasoning: Building on classical logic and philosophical discourse, qualitative claims are scrutinized for overlapping themes, rather than exact numeric agreement.
• Non-Contradiction Principle: Elements that do not explicitly negate one another can be combined to form a broader perspective. Where divergence does occur (e.g., differing recommendations), it may still be possible to expand the discussion in order to provide a valid context for each viewpoint.

Ensemble Reasoning for Hard Facts: The Classical Detective

Ensemble Reasoning mimics the classic human approach to detective work: if three witnesses offer different but overlapping accounts of an event, investigators combine their testimonies to narrow down the most likely version of the truth.

In other words, we are used to the idea that witnesses might be imperfect or partially inaccurate, yet we still combine their accounts to narrow down suspects.

Consider the following case (all text generated by ChatGPT o1).

As demonstrated, AI can use Ensemble Reasoning to mirror the typical detective process – combining partial, flawed observations into a collectively stronger conclusion.

Ensemble Reasoning for Qualitative Knowledge

Similarly to the detective scenario, Ensemble Reasoning can be extended to any context where multiple statements (factual or opinion-based) need to be reconciled. (The following examples were also generated using ChatGPT o1.)

Deciding on a Programming Degree

The following is an example of Ensemble Reasoning used to combine qualitative knowledge that was generated by AI.

Integrating Factual and Qualitative Claims

Now let’s use the same programming-degree question, but this time each LLM answer will include both data and advice, and we will use an AI to merge the individual AI answers.

Summary

Ensemble Reasoning offers a versatile method for reconciling multiple AI outputs—even if each contains inaccuracies or incomplete information. By pairing a formal approach for factual data (constraint satisfaction, set intersection) with a dialectical approach for qualitative or subjective inputs:

1. Contradictions are flagged for deeper investigation.
2. Overlaps provide reinforced confidence in the combined claims.

Whether in detective work or in synthesizing data-driven advice about a career path, Ensemble Reasoning yields a more robust, logically consistent outcome than any single AI response alone could provide.