The Robustness of Voting Methods to Conflicts of Interest

TLDR

• In RetroPGF, badgeholders vote on projects and allocate funding based on project impact. Badgeholders can be subject to competing interests, including conflicts of interest (COI). While this is typically prohibited and must be disclosed, it is not easy to determine whether this happens in the background.
• To address this gap, in this blog post, we compare the robustness of two different voting mechanisms, Quorum + Threshold, and Pairwise, to badgeholders who exhibit COI behavior. We find that Q+T voting exhibits more robustness to COI than Pairwise, in our models of badgeholder behavior for each voting mechanism.
• More generally, robustness to COI is one dimension of performance that should be considered when determining which voting mechanism to use. Other dimensions, such as capital allocation efficiency, are also equally important. Increasing the performance in one dimension will likely result in decreased performance in another. Therefore, choosing the optimal voting mechanism is a nuanced decision that depends on a variety of constraints and goals of the ecosystem.

Introduction

In our first blog post on voting mechanism design, we discussed the COI performance of the pairwise voting mechanism. In this article, we discuss the robustness of the Quorum + Threshold (Q+T) voting mechanism to conflicts of interest (COI), and compare it to the COI performance of the pairwise voting mechanism.

Quorum+Threshold Voting Mechanism

In Q+T voting design, badgeholders are asked to vote on how much funding they would like to allocate to projects. Each badgeholder votes on how much to fund each project and these results are aggregated across all badgeholders. Badgeholders must stay within certain constraints, such as the minimum and maximum funding units they can assign to a project.

Votes are aggregated through some function, such as the mean or median. A particular project is only eligible for funding if it receives enough votes (controlled by the quorum parameter), and, optionally, must also receive a high enough score to qualify for funding (the threshold parameter).

The crucial difference highlighted between Q+T voting and pairwise voting is that in Q+T voting, badgeholders are 1) directly assigning their desired funding amounts to each project and 2) they are making assessments of badgeholder impact across all eligible projects simultaneously. In pairwise, badgeholders make more local decisions by only comparing one project against each other; global rankings are then inferred through an algorithm. More details can be found here.

From a voting design perspective, directly assigning the desired funding amount to each project may be preferred because it enables the badgeholder to directly express their view of the project’s impact on what is essential to projects (funding received). This contrasts with pairwise voting, where projects are ranked but then go through a process of inferring ranks and assigning distributions. Conversely, it may be that ranking projects and then having a complicated process to determine funding amounts reduces the bias inherent in badgeholders’ decision-making when viewing all of the projects from a global perspective.

Experimental Setup To determine the COI performance of the Q+T voting design, we build on the voting_mechanism_design open-source framework and include the design of a new agent that represents a badgeholder that votes in a Q+T voting mechanism with COI behavior. A Q+T voter agent has the following parameters that control how the agent behaves in the simulation:

1. Badgeholder Laziness — This value between 0 and 1 determines how many projects a badgeholder votes on. A higher laziness score means that the badgeholder votes in less projects, and vice versa.
2. Badgeholder COI factor — This value between 0 and 1 determines “how strongly” the COI badgeholder implements their COI behavior. 0 means no COI and 1 means the strongest possible implementation of the COI badgeholder.
3. Badgeholder Expertise — This value determines how correlated the badgeholder’s rankings of projects.

To set the forthcoming results in context, we first define the agent behavior of a Q+T badgeholder.

1. Rank the projects based on their expertise.
2. Vote on the top N projects, based on their configured laziness. The number of projects that they vote on is determined by the following formula: ballot_size = (1-laziness)*num_projects
3. Assign funding amounts to their chosen ballots, according to a linearly decreasing scale.

If a badgeholder is engaging in COI, then the above process is altered as follows:

1. Projects are still initially ranked based on their expertise.
2. The top N projects are modified to include the COI project in the ballot. The COI factor determines how far up the COI project’s ranking is moved (illustrated in the figure below).
3. Funds are assigned on a linearly decreasing scale, as before.

Fig 1 below illustrates how project rankings can be altered. In Step 1, they move their COI project to within the ballot size based on their configured laziness. Then, they move the COI project up by the number of slots proportional to the COI factor.

Fig 1: Q+T COI Badgeholder and how they modify rankings to apply COI.

Results

To measure the effect of COI, we show the change in project ranking with and without COI enabled (after scoring, to include the effects of quorum, threshold, and the scoring function). The simulation framework fixes the random seed so that the probabilistic decisions made by badgeholders are replicated to isolate the effect of COI.

Fig 2 and 3 below show the effect of COI on the change in project rankings, sweeping across the quorum size and the scoring function, respectively.

Sweeping across the quorum size indicates that as the quorum increases, the effect on COI decreases regardless of the particular expertise or laziness distribution of the badgeholder population, although to varying degrees. This is indicated by the reduced change in project ranking as quorum size increases and the number of simulations for which the COI’ed project did not meet the requirements to be considered for funding (indicated by the X above each category). Intuitively, it makes sense that the quorum variable has this effect because increasing the quorum size makes it more difficult for a single badgeholder’s votes to be included in the overall scoring.

The results for sweeping across the scoring functions is shown in Fig 3. For low expertise regimes, mean and quadratic appear to have a lower variance and mean project ranking change than the median, whereas, for higher expertise, the median concretely offers better COI resistance.

Fig 2: The change in project ranking as a function of quorum for different badgeholder laziness and expertise configurations indicated by the legend for a medium impact project, a COI factor of 0.5, using the median score aggregation function. In this simulation, badgeholders are limited in their ability to vote for a project of at most 16 units and a minimum of 1 unit. The total units available for distribution are 100. For each configuration, 32 simulations are run, and the X’s indicate the number of times for each category that projects did not meet the quorum requirement.

Fig 3: The change in project ranking as a function of the aggregation function for different badgeholder laziness and expertise configurations (indicated by the legend) for a medium impact project, a COI factor of 0.5, using a quorum of 5. In this simulation, badgeholders are limited in their ability to vote for a project of at most 16 units and a minimum of 1 unit. The total units available for distribution are 100. For each configuration, 32 simulations are run, and the X’s indicate the number of times for each category that projects did not meet the quorum requirement.

Next, we compare the COI resistance performance of Q+T to Pairwise. A subset of the results presented in the previous blog post are shown in Fig 3. We observe that there is generally a higher variance in the effect of COI on pairwise results than there is for Q+T voting. Additionally, across all subsets of laziness and expertise, the average shift in project ranking is lower for Q+T voting than for pairwise voting, suggesting that Q+T voting, as modeled by the agents above, is more robust to COI than pairwise.

Fig 4: Change in project ranking for pairwise voting, with a COI factor of 0.5 for various combinations of expertise and laziness.

Discussion

The experiments above indicate that Q+T voting is more robust. We caution that while this is true if agents behaved according to the model specified above, badgeholders in real-life may exhibit different behavior, which would directly affect the COI performance of the voting mechanism. Some examples include:

1. Modeling of laziness — laziness, as implemented in the simulations above, determines the ballot size of the badgeholders. It could, however be implemented differently — for example, a lazy badgeholder may choose to distribute their funding evenly amongst all projects (uniform distribution) rather than concentrate their funding on a few projects.
2. Assigning funding — agents in the simulation above assign funding in a linearly decreasing schedule based on their project rankings. However, different schedules are equally valid and will result in different COI performance.
3. Max score allowed for a project — this dampens the effectiveness of an isolated high score and will thus affect how much a particular COI badgeholder will affect the overall results.

Considering the above, it is still appropriate to note that the Q+T voting mechanism has more built-in safeguards to dampen the effect of COI. Specifically, the quorum, threshold, and max_amount parameters all dampen the maximum effect that a particular badgeholder can have on a single project. The pairwise mechanism does not have these, which explains why pairwise performs worse for COI than Q+T. We suggest exploring more robust algorithms to infer rankings from the raw pairwise data to include this dampening effect.

Conclusion

In this post, we described the Q+T voting mechanism and quantitatively analyzed its performance to COI badgeholders. We found that, in general, the Q+T voting mechanism is more robust to COI than the pairwise mechanism. As the quorum size increases or the maximum allowable vote for a particular project decreases, this increases the robustness of a particular badgeholder’s COI. However, it is important to note that these parameter configurations come at a cost — namely, that they also dampen any strong voting patterns that are not COI. For example, reducing the maximum amount that any badgeholder can vote for a single project will serve to flatten the distribution of the overall funding — which may not be desirable to send the signal a particular project is indeed very impactful to the ecosystem.

Parameter setting and tweaking should be done with these considerations in mind, and in practice, it is a complex art.

Additionally, we contend that it is not appropriate to discuss COI performance in isolation of the voting mechanism since it depends as much on the badgeholder behavior. However, the mechanism is also integral to this process because the mechanism’s framework will induce certain behaviors. One does not affect the other, but both affect each other in a loop. An easy way to see this is to consider how COI is implemented in the Q+T badgeholder agent versus the pairwise voting agent. In Q+T, a COI agent moves their ranking of the COI project and then assigns it funding. In Pairwise, the COI agent votes for their COI project a number of times. Fundamentally, there is no way in the pairwise mechanism to precisely affect their ranking of projects through the pairwise mechanism, only approximately through voting.

Building on this, we note that an additional nuance is that while the voting mechanism affects the type of COI behavior and vice versa, artificially normalizing them across voting mechanisms is a modeling choice. Voting mechanisms inherently induce different behavior from badgeholders.

Our next steps are to explore the effect of collusion on Q+T voting. We discussed in this post how parameters like quorum and threshold help dampen the effect of COI, but those dampening effects can be overcome with multiple parties cooperating with each other. Additionally, we are seeking to expand the simulator to other voting mechanisms that can be applied to this style of funding and welcome contributions from others interested in working on this space!

---

This research was conducted by CryptoEconLab as part of our ongoing work in governance mechanism design. For more insights on crypto economics and protocol design, explore our other publications and case studies.