# SentenceTransformers parity

Sting 0.1.0 targets **SentenceTransformers 6.1.0 inference behavior** for a supported
subset of modules and utilities. It is an Elixir API over Bumblebee, Axon, and Nx,
not a replacement for the full Python package. Python is a test dependency only.

## What parity means

Generated-checkpoint tests save tiny Python models, load those same weights in
Sting, feed the same inputs to both implementations, and compare outputs within
explicit floating-point tolerances. Other tests compare utility outputs directly.
The suite pins the SentenceTransformers version and reports Python, Torch, Elixir,
OTP, Nx backend, and Elixir dependency versions.

Passing these cases establishes agreement for the tested paths. It does not
establish compatibility with every checkpoint, architecture, input shape, dtype,
backend, or future SentenceTransformers version. Release validation uses
Nx.BinaryBackend; accelerated backends require their own validation.

## Coverage

| Feature | Evidence | Scope |
| --- | --- | --- |
| Dense Transformer pipelines | Python-generated checkpoints | Pooling, Dense, Normalize, LayerNorm, inference-only Dropout |
| Pooling | Python comparisons | CLS, max, mean, mean-sqrt-length, weighted mean, last-token; padding and fully masked cases |
| Dense activations | Python comparisons | Identity, Tanh, ReLU, GELU, SiLU, Sigmoid, Mish, LeakyReLU |
| Prompts and sequence limits | Python-generated checkpoints and local regressions | Saved/default/named/custom prompts, prompt exclusion, truncation, compiled padding |
| Dense layer fusion | Python comparisons | Consecutive bias-free linear layers; refusal of biased or activated layers |
| Sparse encoders | Python comparisons | MLMTransformer + SpladePooling and SparseStaticEmbedding; dot similarity and search |
| Multi-vector encoders | Generated Python MultiVectorEncoder | Token projection, Normalize, one mask, final hierarchical pooling, query/document prompts and limits |
| Late-interaction scoring | Python comparisons and local regressions | MaxSim/MeanMaxSim, variable lengths, masked padding, fully masked documents |
| Similarity | Python utility comparisons | Cosine, dot, negative Euclidean/Manhattan distance, pairwise variants, pairwise angle |
| Quantization | Python comparisons | Binary/unsigned binary and int8 for tested shapes and calibration; Hamming results checked against Python bit operations |
| Search and mining | Python utility comparisons | Chunked top-k, paraphrase mining, communities, embedding-level hard negatives with tested margins/ranges |
| StaticEmbedding / WeightedLayerPooling | Local fixture tests | Implemented, without Python parity coverage |
| Matryoshka truncation | Local behavior tests | Slicing and optional renormalization, without Python parity coverage |
| Causal-model pooling fallback | Local configuration tests | Last-token selection for recognized causal architectures; not end-to-end parity |
| Cross encoding | Bumblebee delegation | No SentenceTransformers comparison coverage |

Evidence lives in
[dense and utility comparisons](https://github.com/dkuku/sting/blob/v0.1.0/test/sting/python_comparison_test.exs),
[sparse comparisons](https://github.com/dkuku/sting/blob/v0.1.0/test/sting/sparse_test.exs),
[multi-vector comparisons](https://github.com/dkuku/sting/blob/v0.1.0/test/sting/multi_vector_test.exs),
and [mining comparisons](https://github.com/dkuku/sting/blob/v0.1.0/test/sting/mining_test.exs).
The source checkout also contains local regressions in `test/sting_test.exs`,
`test/sting/dense_edge_test.exs`, and `test/sting/utilities_test.exs`.

## API and behavior differences

### Encoding

- Dense and sparse servings return `%{embedding: tensor}` for one string and a
  list of those maps for a list of strings. Stack their `:embedding` values with
  `Nx.stack/1` before passing them to matrix-based retrieval utilities.
- Multi-vector encoding returns a `{tokens, dimensions}` tensor for one string
  or a list of token tensors for multiple strings. MaxSim returns a
  `{query_count, document_count}` score matrix.
- Sparse encodings occupy dense Nx vocabulary tensors. Zeros do not imply sparse
  storage or sparse-kernel acceleration.
- Base-model support follows Bumblebee. Unsupported modules are errors;
  arbitrary Python modules, training, fine-tuning, multimodal inputs, and
  ONNX/OpenVINO exports are outside this release.
- HierarchicalTokenPooling must be the final module and uses host-side clustering.
  Multiple MultiVectorMask modules are rejected. Scoring-mask removal of every
  token during encoding raises an error; explicitly masked scoring tensors are
  supported and tested separately.

### Prompts and padding

Use `Sting.load_tokenizer/2` and `Sting.text_embedding/3` rather than bypassing them
with Bumblebee helpers when saved prompts or limits matter. Query helpers prefer
the `"query"` prompt; corpus helpers try `"document"`, `"passage"`, then `"corpus"`.
Both otherwise use the saved default. An explicit prompt option takes precedence;
`prompt: false` disables the saved prefix.

Saved maximum sequence length controls truncation. Compilation buckets only add
padding, except that their largest size also caps truncation. `:max_seq_length`
is an explicit serving override. Compilation and dynamic tokenization preserve
special-token processing, including EOS-only padding tokenizers.

### Retrieval and quantization

- Mining operates on already-computed embeddings and returns IDs/scores. It does
  not implement Python Dataset output, text/model orchestration, FAISS search,
  cross-encoder rescoring, or multiprocess mining.
- Hard-negative mining expects one positive vector per query and excludes every
  vector-equal corpus entry by default. Text identity is not available at this
  boundary. Filtering applies to the ranked candidate window.
- BinaryBackend search ties prefer lower corpus IDs. Other Nx backends and
  Python/Torch can choose different members at an equal-score top-k cutoff.
- Chunk sizes bound score blocks, not total corpus storage or returned results.
  Community extraction and hierarchical clustering still perform host work.
- Binary quantization pads each last dimension to a multiple of eight. Python
  comparisons include a 12-dimensional input that requires padding. Int8
  supports explicit per-dimension ranges or calibration
  embeddings; reusing calibration is the caller's responsibility.

## Reproduce the comparisons

From the repository root:

```sh
mix deps.get
python3 -m venv .venv
.venv/bin/pip install sentence-transformers==6.1.0 torch numpy datasets
STING_PYTHON="$PWD/.venv/bin/python" mix test --exclude network
```

To include public Hugging Face checkpoints:

```sh
STING_PYTHON="$PWD/.venv/bin/python" mix test
```

An incompatible Python environment fails with setup instructions. `uv` is also
supported when `STING_PYTHON` is unset. The Python parity CI job installs the
pinned SentenceTransformers version and runs local checkpoint comparisons.
Network tests are a separate release check because they depend on external
model availability. Keep the printed version information with parity results;
Torch and other Python dependencies are reported rather than fully locked.