Fix the aggregations and orders a mixed index answers - #5032

Fixes #5031

When a mixed index answers the query in front of count(), min(), max(), sum() or mean(), JanusGraph computes the aggregation in the index. That answer differed from the one the traversal computes from the elements:

• An aggregation of no values gave 0 with Elasticsearch, and 0, NaN or an exception with Lucene and Solr, where TinkerPop has no result.
• Lucene failed on most aggregations once some matching elements lacked the key, and on the minimum and maximum of a LIST or SET key.
• A minimum, maximum, sum or mean ignored a limit() or range() in front, and a count ignored the range's offset.
• The index aggregated without the transaction's own changes, and over a key that isn't enabled in it yet.
• A count stopped at query.hard-max-limit or query.smart-limit.
• A minimum, maximum, sum or mean after a has() on a key which doesn't exist, or one that a composite index answers, failed while the traversal was optimized.

Separately, Lucene, and Solr with JanusGraph's schemas, sorted the elements without the key of an order().by(key) among the others, as 0 for numbers and dates. JanusGraph puts them last when it orders elements itself, and so does Elasticsearch.

The issue has the tables of what master gives on each backend.

Changes

• JanusGraphMixedIndexAggStep:
  • A null answer gives no result.
  • The step only stands in for the traversal where the index gives the same answer:
    • for exactly one mixed index subquery;
    • outside a transaction with uncommitted changes, any of which sends the aggregation to memory, as the index doesn't hold them;
    • for a key which is ENABLED in the index;
    • and, except for a count, without a limit or an offset.
  • A count stops at the traversal's limit: the index query gets that limit instead of the one the query builder sized for fetching elements in batches. The offset of a range() is taken off the count.
  • The Javadoc of IndexProvider.queryAggregation and MixedIndexAggQuery.execute says what null means.
• Lucene:
  • The sort fields of a query's order put the documents without the field after the others in either direction. They sort as the type's greatest value when ascending and its least when descending, and strings without a value sort last.
  • The minimum and maximum of a single-valued field come from the first, in that order, of the matching documents which have a value (a FieldExistsQuery filter). Without the filter, a document without a value could tie with one holding Long.MAX_VALUE and come first.
  • A LIST or SET field has no doc values to sort by, so its minimum and maximum come from StatsCollector, renamed from SumCollector, as every sum and mean does. It reads every stored value of the matching documents and counts them.
  • The mean divides by the number of values, not the number of documents.
• Elasticsearch:
  • RestAggValue.value is a Double, so the null of a min, max or avg aggregation of no values stays null. ElasticSearchClient.avg returns Double.
  • A sum comes from the stats aggregation, whose count tells a sum of no values from a sum of 0.
  • executeAggs returns the parsed aggregation. The line that Bound Elasticsearch counts by their limit and ask aggregations for less #5013 changes is untouched, so the two merge cleanly.
• Solr:
  • The stats requests ask for the count of values as well, and a count of 0 gives null.
  • The test and example schemas declare sortMissingLast="true" on int, long, float, double and date and their t aliases, as their string and boolean types already did.
• Docs:
  • The changelog entry has upgrade notes: next() on an aggregation of no values now throws NoSuchElementException, and the changed signatures.
  • The Solr page has a section on where the schema puts elements without a value.

Not changed:

• A sum keeps the number type the index gives it, a Long for whole numbers and a Double for decimals. TinkerPop keeps the type of the values unless the sum outgrows it. The tests compare numeric values.
• The ordering test follows JanusGraph's own order of elements without the key, last. When TinkerPop's OrderGlobalStep orders, it drops them, as their by() produces nothing. JanusGraph folds an order that a mixed index can answer into its own step, which keeps them, and this PR makes Lucene and Solr put them where that step does.
• A Lucene count beyond 1000 matches can be too low. That is a separate issue, with its own PR.

Tests

• JanusGraphIndexTest.testMixedIndexAggregationsOfTheValuesThereAre (new, every index backend):
  • What it checks:
    • min, max, sum and mean of an Integer, a Double and a LIST key, over vertices with and without values, a match without values, and no match.
    • The minimum of a Long key holding only Long.MAX_VALUE, and the maximum of one holding only Long.MIN_VALUE, each beside a vertex without a value that was indexed first.
    • Counts after limit() and range(), and aggregations after them.
    • Aggregations in a transaction with changes, and over a key added to the index after its data.
    • A count under query.hard-max-limit = 1.
    • A has() on a key which doesn't exist, and one that a composite index answers.
  • Apart from those last two has() cases, every case is compared with the in-memory result of the same traversal and with the expected value, and the profile must show whether JanusGraphMixedIndexAggStep answered it.
  • On master: Lucene fails with NullPointerException at its first aggregation, Elasticsearch gives [0] where memory gives nothing, and Solr fails with NullPointerException.
  • Each of these, reverted alone, fails the test on Lucene: the transaction check, the ENABLED filter, the index query's limit, the count's offset, the limit and offset condition, and the existence filter.
• JanusGraphIndexTest.testMixedIndexOrdersElementsWithoutTheKeyLast (new): Integer, Double and String keys, ascending and descending, with and without limit. On master it fails with Lucene and Solr and passes with Elasticsearch.
• Locally, Java 11:
  • Lucene: BerkeleyLuceneTest (83), LuceneIndexTest (255, 148 skipped) and the module's other tests.
  • Elasticsearch 9.5.4: BerkeleyElasticsearchTest (94, 1 skipped), ElasticsearchIndexTest (287) and the module's unit tests.
  • Solr: the Solr container doesn't start on this machine, master included. A scratch subclass, not part of this PR, ran JanusGraphIndexTest against Solr 9.10.1 in process instead, through SolrRunner's MiniSolrCloudCluster with the test configset. Of its 86 tests, 84 pass. testClearStorage is disabled, as in the Solr test classes, and testOrderingListProperty can't create its collection, which SolrRunner has no configset for.
• Solr 8.11.4 and 9.10.1 in throwaway containers, queried with curl. Both:
  • compute {!min=true max=true sum=true mean=true count=true} over a multi-valued int point field across all values: min -7, max 10, count 3, sum 6, mean 2.0;
  • answer no values with a null min and max, a count of 0, a sum of 0.0 and a mean of NaN;
  • sort the missing values as 0 without sortMissingLast, and last in both directions with it.
• Every case of the issue's tables, run on master and on this branch with each backend: on the branch, all three backends print the same results as in memory.

🤖 Generated with Claude Code