Embedding-level retrieval mining utilities aligned with SentenceTransformers.
The functions accept already-computed embeddings. This keeps retrieval and mining independent of a particular encoder while preserving the selection rules used by SentenceTransformers' utility functions.
Summary
Functions
Finds non-overlapping cosine-similarity communities in embeddings.
Mines hard negatives from embedding-level query, positive, and corpus inputs.
Finds the highest-scoring unique pairs of embeddings.
Functions
Finds non-overlapping cosine-similarity communities in embeddings.
Communities are ordered by decreasing size, and each community starts with
its central (highest-scoring) embedding, matching SentenceTransformers.
:threshold defaults to 0.75, :min_community_size to 10, and
:batch_size to 1_024.
Mines hard negatives from embedding-level query, positive, and corpus inputs.
Each result is %{query_id, positive_id, negative_id, positive_score, negative_score}. Positive corpus entries are excluded by default. The
selection supports SentenceTransformers' :range_min, :range_max,
:min_score, :max_score, :absolute_margin, :relative_margin, and
:sampling_strategy (:top or :random) options. :query_chunk_size
(default 100) and :corpus_chunk_size (default 100_000) bound score
blocks. Vector-equal positives are excluded on the Nx backend before each
block is ranked. :num_negatives defaults to 3; :range_min defaults to
0; :range_max defaults to range_min + num_negatives + 1; and
:sampling_strategy defaults to :top (:random is also supported).
Finds the highest-scoring unique pairs of embeddings.
Results have the shape [[score, first_id, second_id], ...], ordered by
descending score. :query_chunk_size and :corpus_chunk_size bound the
score matrix sizes; :max_pairs caps the returned pairs and :top_k limits
the candidates considered per query per corpus chunk. Defaults are 100,
10_000, 500_000, and 100 respectively; :score_function defaults to
:cos_sim and also accepts :dot_score, :euclidean_sim, :manhattan_sim,
or a two-argument scoring function.