How well does VOLO actually sort?

The test

Take a load order somebody played on and enjoyed. Rebuild the masterlist from scratch with that order left out, so the tool has never seen it. Sort the mods and count how many pairs come out in the same relative order as the one the player actually used. Shuffle the same list at random for comparison.

Leaving the order out matters. Scored against orders it learned from, VOLO looks about four points better than it is, because it is being asked to recall an answer it has already read.

The result

Across 106 working orders, VOLO agrees with the player 67.7 percent of the time, against 50.8 percent for a random shuffle. So it is doing real work, and it is nowhere near a solved problem.

The corpus holds 153 working orders and 106 are scored here. 45 of them VOLO sorted itself, and marking its own homework measures nothing except that the sort is deterministic, and two more are byte-identical copies of other files.

The average treats every order equally, so a 22-mod list counts as much as a 1,647-mod one. Weighted by the number of mods, which is closer to what someone with a real load order experiences, it comes out at 68.0 percent.

Where it helped most

The five orders VOLO reconstructed best, by how far it beat a random shuffle of the same list. This is the good end of the range and nothing more: the average above also carries two orders where the sort did no better than chance at all.

The ten most recent

Newest submissions first, which is the part of the corpus that has had the least chance to influence the rules it is being scored against.

Things that sounded sensible and were not

Most of what was tried made the sort worse. Recording the failures is the only way to stop them being tried again. Each figure below is against the baseline of its own run, from an older scoring pass, so the pairs compare with each other and not with the headline above; where only a cost in points is given, that cost is against the same run it was measured in.

Filing unknown mods near their neighbours

Instead of sending an unplaced mod to the end, give it the position of the last known mod above it. Scored 61.8 against 63.6. Mods nobody has categorised really do pile up at the end of real load orders, so the tool leaves them there.

Adopting the divider order wholesale

Throwing out the learned sequence and using the community divider numbering for everything scored 60.4 against 63.6.

Reading categories out of divider names

The divider set names a hundred things the categories do not, so using those labels to place otherwise unreachable mods looked free. It placed 67 mods and cost 0.7 points.

Trusting the mod's own listing

Using the Nexus or mod.io category for mods the community has never placed cost 0.6 points, and it was rejected on that basis. Then it was adopted anyway. The measurement was answering the wrong question, which is the next section.

Why the score is not the goal

The test scores VOLO against orders that already work. That quietly rewards doing nothing: leave every mod exactly where the player put it and you score perfectly, while sorting nothing at all. Any real change costs points before it earns them back.

It also flatters ignorance. A mod dumped at the end because the tool knows nothing about it scores well, precisely because unplaced mods cluster at the end anyway. That says something about the corpus, not about whether the sort is useful to you.

So when the community divider structure was adopted, the score fell from 60.3 to 57.5 and it was adopted regardless. Mods on a known position went from 967 of 3,008 to 2,662, and mods with no category at all fell from 650 to 338. The exported order gained headings a player can read. That was judged worth more than three points on a metric fitted to nine files.

The limits

106 scored orders is a small corpus, and it is the thing holding the tool back rather than the sorting itself. The headline carries about 1.1 points of uncertainty on that sample, and it would take something like 510 scored orders to bring that under half a point.

Orders also arrive anonymously, which is deliberate but costs something here: there is no way to know how many different people are behind them, so what reads as 106 independent opinions may be fewer. Every measurement on this page should be read with that in mind.

The automated tests parse VOLO's own output with VOLO's own parser, which proves the tool is consistent with itself and nothing more. Exported orders have been loaded in Baldur's Gate 3 by hand and worked, but that is a manual check, not something the test suite can promise you.

If you have played on an order, submitting it moves these numbers more than any change to the code would.