EmbeddingGemma vs multilingual-e5-large para embeddings
Um modelo de embeddings multilíngue minúsculo e um médio
Fatos do catálogo revisado
| Especificação | EmbeddingGemma (300M) | Multilingual E5 Large Instruct |
|---|---|---|
| Tarefa | Embeddings e busca | Embeddings e busca |
| Parâmetros | 303M | 560M |
| Licença | Termos do publicador: Gemma Terms of Use | MIT |
| Uso comercial | Permitidotexto da licença | Permitidotexto da licença |
| Qualidade · MTEB (Multilingual, v2) | 61,14768d · execução independente · fonte | 63,221.024d · execução independente · fonte |
| Memória | estimativa de embeddings de 8 bits (~3 GB) | estimativa de embeddings de 8 bits (~3 GB) |
| Checkpoint | ficha oficial | ficha oficial |
No MTEB (Multilingual, v2), o Multilingual E5 Large Instruct tem 63,22 e o EmbeddingGemma (300M) 61,14, 2,1 pontos de diferença na mesma versão do benchmark. Execuções independentes do conjunto público de resultados do MTEB, mesma versão do benchmark para os dois modelos. Escala: média das tarefas, 0 a 100, maior é melhor.
Memória em três máquinas
Estimativas de embeddings de 8 bits: pesos mais 20% de folga e reserva fixa de 2 GB, então o valor é o mesmo em todas as máquinas. Estimativas de planejamento, não execuções medidas.
| Máquina | EmbeddingGemma (300M) | Multilingual E5 Large Instruct |
|---|---|---|
| RTX 4060 · 8 GB de VRAM | ~3 GBCabe na memória rápida | ~3 GBCabe na memória rápida |
| MacBook Air 13 polegadas M4 · 16 GB unificados | ~3 GBCabe na memória rápida | ~3 GBCabe na memória rápida |
| Mac mini M4 · 16 GB unificados | ~3 GBCabe na memória rápida | ~3 GBCabe na memória rápida |
EmbeddingGemma (300M): estimativa de embeddings de 8 bits (~3 GB). Multilingual E5 Large Instruct: estimativa de embeddings de 8 bits (~3 GB). Fórmulas
Tudo o que roda em cada máquina: RTX 4060 · 8 GB de VRAM · MacBook Air 13 polegadas M4 · 16 GB unificados · Mac mini M4 · 16 GB unificados
Qual escolher
O EmbeddingGemma tem 0,3 bilhão de parâmetros e vetores de 768 dimensões; o Multilingual E5 Large Instruct tem 0,56 bilhão e 1.024 dimensões. Em execuções independentes do MTEB (Multilingual, v2) o E5 Large Instruct tem nota maior, e o EmbeddingGemma tem cerca da metade do tamanho, o que pesa em celulares e máquinas pequenas. O EmbeddingGemma lê até 2.048 tokens e o E5 até 512, então trechos longos precisam ser mais divididos no E5. O E5 é MIT; o EmbeddingGemma usa os Termos de Uso do Gemma, que permitem uso comercial.
Arquivos e comandos para rodar
O arquivo padrão da fonte coletada mais baixada, ou os comandos que cada ficha documenta. Não executados por este site.
EmbeddingGemma (300M)
Formas documentadas de rodar
Não foi possível ler a ficha do modelo (ausente ou restrita)
Multilingual E5 Large Instruct
Formas documentadas de rodar
Copiado como aparece na fonte vinculada (lida em 2026-10-18); não executado por este site.
from sentence_transformers import SentenceTransformer
def get_detailed_instruct(task_description: str, query: str) -> str:
return f'Instruct: {task_description}\nQuery: {query}'
# Each query must come with a one-sentence instruction that describes the task
task = 'Given a web search query, retrieve relevant passages that answer the query'
queries = [
get_detailed_instruct(task, 'how much protein should a female eat'),
get_detailed_instruct(task, '南瓜的家常做法')
]
# No need to add instruction for retrieval documents
documents = [
"As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.",
"1.清炒南瓜丝 原料:嫩南瓜半个 调料:葱、盐、白糖、鸡精 做法: 1、南瓜用刀薄薄的削去表面一层皮,用勺子刮去瓤 2、擦成细丝(没有擦菜板就用刀慢慢切成细丝) 3、锅烧热放油,入葱花煸出香味 4、入南瓜丝快速翻炒一分钟左右,放盐、一点白糖和鸡精调味出锅 2.香葱炒南瓜 原料:南瓜1只 调料:香葱、蒜末、橄榄油、盐 做法: 1、将南瓜去皮,切成片 2、油锅8成热后,将蒜末放入爆香 3、爆香后,将南瓜片放入,翻炒 4、在翻炒的同时,可以不时地往锅里加水,但不要太多 5、放入盐,炒匀 6、南瓜差不多软和绵了之后,就可以关火 7、撒入香葱,即可出锅"
]
input_texts = queries + documents
model = SentenceTransformer('intfloat/multilingual-e5-large-instruct')
embeddings = model.encode(input_texts, convert_to_tensor=True, normalize_embeddings=True)
scores = (embeddings[:2] @ embeddings[2:].T) * 100
print(scores.tolist())
# [[91.92853546142578, 67.5802993774414], [70.38143157958984, 92.13307189941406]]import torch.nn.functional as F
from torch import Tensor
from transformers import AutoTokenizer, AutoModel
def average_pool(last_hidden_states: Tensor,
attention_mask: Tensor) -> Tensor:
last_hidden = last_hidden_states.masked_fill(~attention_mask[..., None].bool(), 0.0)
return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None]
def get_detailed_instruct(task_description: str, query: str) -> str:
return f'Instruct: {task_description}\nQuery: {query}'
# Each query must come with a one-sentence instruction that describes the task
task = 'Given a web search query, retrieve relevant passages that answer the query'
queries = [
get_detailed_instruct(task, 'how much protein should a female eat'),
get_detailed_instruct(task, '南瓜的家常做法')
]
# No need to add instruction for retrieval documents
documents = [
"As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.",
"1.清炒南瓜丝 原料:嫩南瓜半个 调料:葱、盐、白糖、鸡精 做法: 1、南瓜用刀薄薄的削去表面一层皮,用勺子刮去瓤 2、擦成细丝(没有擦菜板就用刀慢慢切成细丝) 3、锅烧热放油,入葱花煸出香味 4、入南瓜丝快速翻炒一分钟左右,放盐、一点白糖和鸡精调味出锅 2.香葱炒南瓜 原料:南瓜1只 调料:香葱、蒜末、橄榄油、盐 做法: 1、将南瓜去皮,切成片 2、油锅8成热后,将蒜末放入爆香 3、爆香后,将南瓜片放入,翻炒 4、在翻炒的同时,可以不时地往锅里加水,但不要太多 5、放入盐,炒匀 6、南瓜差不多软和绵了之后,就可以关火 7、撒入香葱,即可出锅"
]
input_texts = queries + documents
tokenizer = AutoTokenizer.from_pretrained('intfloat/multilingual-e5-large-instruct')
model = AutoModel.from_pretrained('intfloat/multilingual-e5-large-instruct')
# Tokenize the input texts
batch_dict = tokenizer(input_texts, max_length=512, padding=True, truncation=True, return_tensors='pt')
outputs = model(**batch_dict)
embeddings = average_pool(outputs.last_hidden_state, batch_dict['attention_mask'])
# normalize embeddings
embeddings = F.normalize(embeddings, p=2, dim=1)
scores = (embeddings[:2] @ embeddings[2:].T) * 100
print(scores.tolist())
# => [[91.92852783203125, 67.580322265625], [70.3814468383789, 92.1330795288086]]Números que cada publicador informa
Os resultados de cada ficha, com a configuração de cada uma. Publicadores escolhem testes e configurações diferentes, então estes valores nunca são comparados entre publicadores aqui.
EmbeddingGemma (300M) · informado por Google
- Mean (Task)
- 61,15MTEB (Multilingual, v2)
- Mean (TaskType)
- 54,31MTEB (Multilingual, v2)
- Mean (Task)
- 69,67MTEB (English, v2)
- Mean (TaskType)
- 65,11MTEB (English, v2)
- Mean (Task)
- 68,76MTEB (Code, v1)
- Mean (TaskType)
- 68,76MTEB (Code, v1)
Valores como impressos na ficha, com a configuração do próprio publicador (seção da ficha: Evaluation › Benchmark Results › Full Precision Checkpoint); coluna “768d”. Não comparáveis entre publicadores e fora de qualquer ranking. Ficha do modelo
Multilingual E5 Large Instruct · informado por intfloat
- MTEB AmazonCounterfactualClassification (en) (en; test; accuracy)
- 76,239Classification
- MTEB AmazonCounterfactualClassification (de) (de; test; accuracy)
- 66,713Classification
- MTEB AmazonCounterfactualClassification (en-ext) (en-ext; test; accuracy)
- 73,853Classification
- MTEB AmazonCounterfactualClassification (ja) (ja; test; accuracy)
- 76,049Classification
- MTEB AmazonPolarityClassification (test; accuracy)
- 96,287Classification
- MTEB AmazonReviewsClassification (en) (en; test; accuracy)
- 56,716Classification
- MTEB AmazonReviewsClassification (de) (de; test; accuracy)
- 53,000Classification
- MTEB AmazonReviewsClassification (es) (es; test; accuracy)
- 48,806Classification
Todos os resultados informados
- MTEB AmazonReviewsClassification (fr) (fr; test; accuracy)
- 48,508Classification
- MTEB AmazonReviewsClassification (ja) (ja; test; accuracy)
- 47,710Classification
- MTEB AmazonReviewsClassification (zh) (zh; test; accuracy)
- 44,662Classification
- MTEB ArguAna (test; ndcg_at_10)
- 58,384Retrieval
- MTEB ArxivClusteringP2P (test; v_measure)
- 46,404Clustering
- MTEB ArxivClusteringS2S (test; v_measure)
- 40,486Clustering
- MTEB AskUbuntuDupQuestions (test; map)
- 63,889Reranking
- MTEB BIOSSES (test; cos_sim_spearman)
- 86,962STS
- MTEB BUCC (de-en) (de-en; test; f1)
- 99,607BitextMining
- MTEB BUCC (fr-en) (fr-en; test; f1)
- 99,145BitextMining
- MTEB BUCC (ru-en) (ru-en; test; f1)
- 97,866BitextMining
- MTEB BUCC (zh-en) (zh-en; test; f1)
- 99,386BitextMining
- MTEB Banking77Classification (test; accuracy)
- 85,734Classification
- MTEB BiorxivClusteringP2P (test; v_measure)
- 40,935Clustering
- MTEB BiorxivClusteringS2S (test; v_measure)
- 36,276Clustering
- MTEB CQADupstackRetrieval (test; ndcg_at_10)
- 42,707Retrieval
- MTEB ClimateFEVER (test; ndcg_at_10)
- 29,858Retrieval
- MTEB DBPedia (test; ndcg_at_10)
- 38,36Retrieval
- MTEB EmotionClassification (test; accuracy)
- 51,51Classification
- MTEB FEVER (test; ndcg_at_10)
- 77,988Retrieval
- MTEB FiQA2018 (test; ndcg_at_10)
- 47,712Retrieval
- MTEB HotpotQA (test; ndcg_at_10)
- 69,317Retrieval
- MTEB ImdbClassification (test; accuracy)
- 94,601Classification
- MTEB MSMARCO (dev; ndcg_at_10)
- 40,43Retrieval
- MTEB MTOPDomainClassification (en) (en; test; accuracy)
- 93,926Classification
- MTEB MTOPDomainClassification (de) (de; test; accuracy)
- 92,677Classification
- MTEB MTOPDomainClassification (es) (es; test; accuracy)
- 92,268Classification
- MTEB MTOPDomainClassification (fr) (fr; test; accuracy)
- 89,966Classification
- MTEB MTOPDomainClassification (hi) (hi; test; accuracy)
- 90,025Classification
- MTEB MTOPDomainClassification (th) (th; test; accuracy)
- 87,555Classification
- MTEB MTOPIntentClassification (en) (en; test; accuracy)
- 82,462Classification
- MTEB MTOPIntentClassification (de) (de; test; accuracy)
- 79,372Classification
- MTEB MTOPIntentClassification (es) (es; test; accuracy)
- 80,340Classification
- MTEB MTOPIntentClassification (fr) (fr; test; accuracy)
- 76,721Classification
- MTEB MTOPIntentClassification (hi) (hi; test; accuracy)
- 77,870Classification
- MTEB MTOPIntentClassification (th) (th; test; accuracy)
- 77,736Classification
- MTEB MassiveIntentClassification (af) (af; test; accuracy)
- 67,663Classification
- MTEB MassiveIntentClassification (am) (am; test; accuracy)
- 60,713Classification
- MTEB MassiveIntentClassification (ar) (ar; test; accuracy)
- 63,611Classification
- MTEB MassiveIntentClassification (az) (az; test; accuracy)
- 69,001Classification
- MTEB MassiveIntentClassification (bn) (bn; test; accuracy)
- 68,541Classification
- MTEB MassiveIntentClassification (cy) (cy; test; accuracy)
- 63,033Classification
- MTEB MassiveIntentClassification (da) (da; test; accuracy)
- 71,809Classification
- MTEB MassiveIntentClassification (de) (de; test; accuracy)
- 72,471Classification
- MTEB MassiveIntentClassification (el) (el; test; accuracy)
- 71,237Classification
- MTEB MassiveIntentClassification (en) (en; test; accuracy)
- 77,058Classification
- MTEB MassiveIntentClassification (es) (es; test; accuracy)
- 72,821Classification
- MTEB MassiveIntentClassification (fa) (fa; test; accuracy)
- 74,254Classification
- MTEB MassiveIntentClassification (fi) (fi; test; accuracy)
- 72,286Classification
- MTEB MassiveIntentClassification (fr) (fr; test; accuracy)
- 73,315Classification
- MTEB MassiveIntentClassification (he) (he; test; accuracy)
- 70,215Classification
- MTEB MassiveIntentClassification (hi) (hi; test; accuracy)
- 71,584Classification
- MTEB MassiveIntentClassification (hu) (hu; test; accuracy)
- 71,923Classification
- MTEB MassiveIntentClassification (hy) (hy; test; accuracy)
- 68,073Classification
- MTEB MassiveIntentClassification (id) (id; test; accuracy)
- 72,623Classification
- MTEB MassiveIntentClassification (is) (is; test; accuracy)
- 65,770Classification
- MTEB MassiveIntentClassification (it) (it; test; accuracy)
- 73,453Classification
- MTEB MassiveIntentClassification (ja) (ja; test; accuracy)
- 74,691Classification
- MTEB MassiveIntentClassification (jv) (jv; test; accuracy)
- 63,036Classification
- MTEB MassiveIntentClassification (ka) (ka; test; accuracy)
- 58,914Classification
- MTEB MassiveIntentClassification (km) (km; test; accuracy)
- 54,428Classification
- MTEB MassiveIntentClassification (kn) (kn; test; accuracy)
- 66,335Classification
- MTEB MassiveIntentClassification (ko) (ko; test; accuracy)
- 70,585Classification
- MTEB MassiveIntentClassification (lv) (lv; test; accuracy)
- 69,112Classification
- MTEB MassiveIntentClassification (ml) (ml; test; accuracy)
- 69,701Classification
- MTEB MassiveIntentClassification (mn) (mn; test; accuracy)
- 66,439Classification
- MTEB MassiveIntentClassification (ms) (ms; test; accuracy)
- 70,800Classification
- MTEB MassiveIntentClassification (my) (my; test; accuracy)
- 64,788Classification
- MTEB MassiveIntentClassification (nb) (nb; test; accuracy)
- 71,658Classification
- MTEB MassiveIntentClassification (nl) (nl; test; accuracy)
- 74,425Classification
- MTEB MassiveIntentClassification (pl) (pl; test; accuracy)
- 73,648Classification
- MTEB MassiveIntentClassification (pt) (pt; test; accuracy)
- 73,628Classification
- MTEB MassiveIntentClassification (ro) (ro; test; accuracy)
- 71,886Classification
- MTEB MassiveIntentClassification (ru) (ru; test; accuracy)
- 74,159Classification
- MTEB MassiveIntentClassification (sl) (sl; test; accuracy)
- 69,963Classification
- MTEB MassiveIntentClassification (sq) (sq; test; accuracy)
- 69,496Classification
- MTEB MassiveIntentClassification (sv) (sv; test; accuracy)
- 74,681Classification
- MTEB MassiveIntentClassification (sw) (sw; test; accuracy)
- 63,006Classification
- MTEB MassiveIntentClassification (ta) (ta; test; accuracy)
- 66,910Classification
- MTEB MassiveIntentClassification (te) (te; test; accuracy)
- 67,623Classification
- MTEB MassiveIntentClassification (th) (th; test; accuracy)
- 69,509Classification
- MTEB MassiveIntentClassification (tl) (tl; test; accuracy)
- 69,307Classification
- MTEB MassiveIntentClassification (tr) (tr; test; accuracy)
- 72,243Classification
- MTEB MassiveIntentClassification (ur) (ur; test; accuracy)
- 67,498Classification
- MTEB MassiveIntentClassification (vi) (vi; test; accuracy)
- 71,288Classification
- MTEB MassiveIntentClassification (zh-CN) (zh-CN; test; accuracy)
- 73,315Classification
- MTEB MassiveIntentClassification (zh-TW) (zh-TW; test; accuracy)
- 69,378Classification
- MTEB MassiveScenarioClassification (af) (af; test; accuracy)
- 73,336Classification
- MTEB MassiveScenarioClassification (am) (am; test; accuracy)
- 65,844Classification
- MTEB MassiveScenarioClassification (ar) (ar; test; accuracy)
- 69,765Classification
- MTEB MassiveScenarioClassification (az) (az; test; accuracy)
- 72,024Classification
- MTEB MassiveScenarioClassification (bn) (bn; test; accuracy)
- 72,757Classification
- MTEB MassiveScenarioClassification (cy) (cy; test; accuracy)
- 68,023Classification
- MTEB MassiveScenarioClassification (da) (da; test; accuracy)
- 77,609Classification
- MTEB MassiveScenarioClassification (de) (de; test; accuracy)
- 77,683Classification
- MTEB MassiveScenarioClassification (el) (el; test; accuracy)
- 76,133Classification
- MTEB MassiveScenarioClassification (en) (en; test; accuracy)
- 80,474Classification
- MTEB MassiveScenarioClassification (es) (es; test; accuracy)
- 76,967Classification
- MTEB MassiveScenarioClassification (fa) (fa; test; accuracy)
- 78,097Classification
- MTEB MassiveScenarioClassification (fi) (fi; test; accuracy)
- 75,208Classification
- MTEB MassiveScenarioClassification (fr) (fr; test; accuracy)
- 77,075Classification
- MTEB MassiveScenarioClassification (he) (he; test; accuracy)
- 73,534Classification
- MTEB MassiveScenarioClassification (hi) (hi; test; accuracy)
- 75,753Classification
- MTEB MassiveScenarioClassification (hu) (hu; test; accuracy)
- 77,095Classification
- MTEB MassiveScenarioClassification (hy) (hy; test; accuracy)
- 71,076Classification
- MTEB MassiveScenarioClassification (id) (id; test; accuracy)
- 77,105Classification
- MTEB MassiveScenarioClassification (is) (is; test; accuracy)
- 71,258Classification
- MTEB MassiveScenarioClassification (it) (it; test; accuracy)
- 77,081Classification
- MTEB MassiveScenarioClassification (ja) (ja; test; accuracy)
- 79,354Classification
- MTEB MassiveScenarioClassification (jv) (jv; test; accuracy)
- 68,423Classification
- MTEB MassiveScenarioClassification (ka) (ka; test; accuracy)
- 66,157Classification
- MTEB MassiveScenarioClassification (km) (km; test; accuracy)
- 60,114Classification
- MTEB MassiveScenarioClassification (kn) (kn; test; accuracy)
- 71,247Classification
- MTEB MassiveScenarioClassification (ko) (ko; test; accuracy)
- 76,463Classification
- MTEB MassiveScenarioClassification (lv) (lv; test; accuracy)
- 73,248Classification
- MTEB MassiveScenarioClassification (ml) (ml; test; accuracy)
- 74,116Classification
- MTEB MassiveScenarioClassification (mn) (mn; test; accuracy)
- 70,017Classification
- MTEB MassiveScenarioClassification (ms) (ms; test; accuracy)
- 74,412Classification
- MTEB MassiveScenarioClassification (my) (my; test; accuracy)
- 68,396Classification
- MTEB MassiveScenarioClassification (nb) (nb; test; accuracy)
- 77,209Classification
- MTEB MassiveScenarioClassification (nl) (nl; test; accuracy)
- 78,524Classification
- MTEB MassiveScenarioClassification (pl) (pl; test; accuracy)
- 76,691Classification
- MTEB MassiveScenarioClassification (pt) (pt; test; accuracy)
- 77,122Classification
- MTEB MassiveScenarioClassification (ro) (ro; test; accuracy)
- 75,259Classification
- MTEB MassiveScenarioClassification (ru) (ru; test; accuracy)
- 77,707Classification
- MTEB MassiveScenarioClassification (sl) (sl; test; accuracy)
- 74,835Classification
- MTEB MassiveScenarioClassification (sq) (sq; test; accuracy)
- 74,718Classification
- MTEB MassiveScenarioClassification (sv) (sv; test; accuracy)
- 79,065Classification
- MTEB MassiveScenarioClassification (sw) (sw; test; accuracy)
- 67,919Classification
- MTEB MassiveScenarioClassification (ta) (ta; test; accuracy)
- 70,931Classification
- MTEB MassiveScenarioClassification (te) (te; test; accuracy)
- 72,408Classification
- MTEB MassiveScenarioClassification (th) (th; test; accuracy)
- 75,182Classification
- MTEB MassiveScenarioClassification (tl) (tl; test; accuracy)
- 72,865Classification
- MTEB MassiveScenarioClassification (tr) (tr; test; accuracy)
- 76,469Classification
- MTEB MassiveScenarioClassification (ur) (ur; test; accuracy)
- 71,893Classification
- MTEB MassiveScenarioClassification (vi) (vi; test; accuracy)
- 74,751Classification
- MTEB MassiveScenarioClassification (zh-CN) (zh-CN; test; accuracy)
- 78,695Classification
- MTEB MassiveScenarioClassification (zh-TW) (zh-TW; test; accuracy)
- 75,457Classification
- MTEB MedrxivClusteringP2P (test; v_measure)
- 36,927Clustering
- MTEB MedrxivClusteringS2S (test; v_measure)
- 35,535Clustering
- MTEB MindSmallReranking (test; map)
- 33,085Reranking
- MTEB NFCorpus (test; ndcg_at_10)
- 35,534Retrieval
- MTEB NQ (test; ndcg_at_10)
- 57,754Retrieval
- MTEB QuoraRetrieval (test; ndcg_at_10)
- 89,149Retrieval
- MTEB RedditClustering (test; v_measure)
- 56,598Clustering
- MTEB RedditClusteringP2P (test; v_measure)
- 64,274Clustering
- MTEB SCIDOCS (test; ndcg_at_10)
- 18,723Retrieval
- MTEB SICK-R (test; cos_sim_spearman)
- 81,732STS
- MTEB STS12 (test; cos_sim_spearman)
- 82,571STS
- MTEB STS13 (test; cos_sim_spearman)
- 87,150STS
- MTEB STS14 (test; cos_sim_spearman)
- 84,971STS
- MTEB STS15 (test; cos_sim_spearman)
- 91,051STS
- MTEB STS16 (test; cos_sim_spearman)
- 87,314STS
- MTEB STS17 (en-en) (en-en; test; cos_sim_spearman)
- 90,028STS
- MTEB STS22 (en) (en; test; cos_sim_spearman)
- 67,630STS
- MTEB STSBenchmark (test; cos_sim_spearman)
- 88,379STS
- MTEB SciDocsRR (test; map)
- 85,872Reranking
- MTEB SciFact (test; ndcg_at_10)
- 71,852Retrieval
- MTEB SprintDuplicateQuestions (test; cos_sim_ap)
- 91,177PairClassification
- MTEB SprintDuplicateQuestions (test; max_ap)
- 91,224PairClassification
- MTEB StackExchangeClustering (test; v_measure)
- 66,851Clustering
- MTEB StackExchangeClusteringP2P (test; v_measure)
- 42,461Clustering
- MTEB StackOverflowDupQuestions (test; map)
- 51,445Reranking
- MTEB SummEval (test; cos_sim_spearman)
- 30,385Summarization
- MTEB TRECCOVID (test; ndcg_at_10)
- 82Retrieval
- MTEB Tatoeba (sqi-eng) (sqi-eng; test; f1)
- 96,75BitextMining
- MTEB Tatoeba (fry-eng) (fry-eng; test; f1)
- 82,466BitextMining
- MTEB Tatoeba (kur-eng) (kur-eng; test; f1)
- 77,333BitextMining
- MTEB Tatoeba (tur-eng) (tur-eng; test; f1)
- 98,267BitextMining
- MTEB Tatoeba (deu-eng) (deu-eng; test; f1)
- 99,333BitextMining
- MTEB Tatoeba (nld-eng) (nld-eng; test; f1)
- 97,2BitextMining
- MTEB Tatoeba (ron-eng) (ron-eng; test; f1)
- 97,183BitextMining
- MTEB Tatoeba (ang-eng) (ang-eng; test; f1)
- 72,811BitextMining
- MTEB Tatoeba (ido-eng) (ido-eng; test; f1)
- 91,917BitextMining
- MTEB Tatoeba (jav-eng) (jav-eng; test; f1)
- 85,276BitextMining
- MTEB Tatoeba (isl-eng) (isl-eng; test; f1)
- 95BitextMining
- MTEB Tatoeba (slv-eng) (slv-eng; test; f1)
- 91,908BitextMining
- MTEB Tatoeba (cym-eng) (cym-eng; test; f1)
- 89,971BitextMining
- MTEB Tatoeba (kaz-eng) (kaz-eng; test; f1)
- 87,722BitextMining
- MTEB Tatoeba (est-eng) (est-eng; test; f1)
- 88,03BitextMining
- MTEB Tatoeba (heb-eng) (heb-eng; test; f1)
- 91,457BitextMining
- MTEB Tatoeba (gla-eng) (gla-eng; test; f1)
- 78,326BitextMining
- MTEB Tatoeba (mar-eng) (mar-eng; test; f1)
- 92,583BitextMining
- MTEB Tatoeba (lat-eng) (lat-eng; test; f1)
- 74,505BitextMining
- MTEB Tatoeba (bel-eng) (bel-eng; test; f1)
- 95,55BitextMining
- MTEB Tatoeba (pms-eng) (pms-eng; test; f1)
- 77,985BitextMining
- MTEB Tatoeba (gle-eng) (gle-eng; test; f1)
- 84,992BitextMining
- MTEB Tatoeba (pes-eng) (pes-eng; test; f1)
- 94,567BitextMining
- MTEB Tatoeba (nob-eng) (nob-eng; test; f1)
- 98,2BitextMining
- MTEB Tatoeba (bul-eng) (bul-eng; test; f1)
- 94,383BitextMining
- MTEB Tatoeba (cbk-eng) (cbk-eng; test; f1)
- 84,104BitextMining
- MTEB Tatoeba (hun-eng) (hun-eng; test; f1)
- 94,333BitextMining
- MTEB Tatoeba (uig-eng) (uig-eng; test; f1)
- 86,820BitextMining
- MTEB Tatoeba (rus-eng) (rus-eng; test; f1)
- 93,567BitextMining
- MTEB Tatoeba (spa-eng) (spa-eng; test; f1)
- 98,6BitextMining
- MTEB Tatoeba (hye-eng) (hye-eng; test; f1)
- 93,518BitextMining
- MTEB Tatoeba (tel-eng) (tel-eng; test; f1)
- 96,083BitextMining
- MTEB Tatoeba (afr-eng) (afr-eng; test; f1)
- 95,400BitextMining
- MTEB Tatoeba (mon-eng) (mon-eng; test; f1)
- 95,492BitextMining
- MTEB Tatoeba (arz-eng) (arz-eng; test; f1)
- 81,859BitextMining
- MTEB Tatoeba (hrv-eng) (hrv-eng; test; f1)
- 96,75BitextMining
- MTEB Tatoeba (nov-eng) (nov-eng; test; f1)
- 82,996BitextMining
- MTEB Tatoeba (gsw-eng) (gsw-eng; test; f1)
- 61,897BitextMining
- MTEB Tatoeba (nds-eng) (nds-eng; test; f1)
- 86,32BitextMining
- MTEB Tatoeba (ukr-eng) (ukr-eng; test; f1)
- 94,783BitextMining
- MTEB Tatoeba (uzb-eng) (uzb-eng; test; f1)
- 80,545BitextMining
- MTEB Tatoeba (lit-eng) (lit-eng; test; f1)
- 92,013BitextMining
- MTEB Tatoeba (ina-eng) (ina-eng; test; f1)
- 96,267BitextMining
- MTEB Tatoeba (lfn-eng) (lfn-eng; test; f1)
- 80,678BitextMining
- MTEB Tatoeba (zsm-eng) (zsm-eng; test; f1)
- 96,483BitextMining
- MTEB Tatoeba (ita-eng) (ita-eng; test; f1)
- 94,667BitextMining
- MTEB Tatoeba (cmn-eng) (cmn-eng; test; f1)
- 96,367BitextMining
- MTEB Tatoeba (lvs-eng) (lvs-eng; test; f1)
- 92,807BitextMining
- MTEB Tatoeba (glg-eng) (glg-eng; test; f1)
- 96,223BitextMining
- MTEB Tatoeba (ceb-eng) (ceb-eng; test; f1)
- 70,782BitextMining
- MTEB Tatoeba (bre-eng) (bre-eng; test; f1)
- 32,938BitextMining
- MTEB Tatoeba (ben-eng) (ben-eng; test; f1)
- 89,133BitextMining
- MTEB Tatoeba (swg-eng) (swg-eng; test; f1)
- 77,679BitextMining
- MTEB Tatoeba (arq-eng) (arq-eng; test; f1)
- 63,070BitextMining
- MTEB Tatoeba (kab-eng) (kab-eng; test; f1)
- 57,848BitextMining
- MTEB Tatoeba (fra-eng) (fra-eng; test; f1)
- 95,017BitextMining
- MTEB Tatoeba (por-eng) (por-eng; test; f1)
- 94,907BitextMining
- MTEB Tatoeba (tat-eng) (tat-eng; test; f1)
- 84,613BitextMining
- MTEB Tatoeba (oci-eng) (oci-eng; test; f1)
- 71,907BitextMining
- MTEB Tatoeba (pol-eng) (pol-eng; test; f1)
- 97,267BitextMining
- MTEB Tatoeba (war-eng) (war-eng; test; f1)
- 74,396BitextMining
- MTEB Tatoeba (aze-eng) (aze-eng; test; f1)
- 93,790BitextMining
- MTEB Tatoeba (vie-eng) (vie-eng; test; f1)
- 97,1BitextMining
- MTEB Tatoeba (nno-eng) (nno-eng; test; f1)
- 94,257BitextMining
- MTEB Tatoeba (cha-eng) (cha-eng; test; f1)
- 51,462BitextMining
- MTEB Tatoeba (mhr-eng) (mhr-eng; test; f1)
- 16,918BitextMining
- MTEB Tatoeba (dan-eng) (dan-eng; test; f1)
- 95,3BitextMining
- MTEB Tatoeba (ell-eng) (ell-eng; test; f1)
- 95,117BitextMining
- MTEB Tatoeba (amh-eng) (amh-eng; test; f1)
- 87,143BitextMining
- MTEB Tatoeba (pam-eng) (pam-eng; test; f1)
- 19,631BitextMining
- MTEB Tatoeba (hsb-eng) (hsb-eng; test; f1)
- 79,452BitextMining
- MTEB Tatoeba (srp-eng) (srp-eng; test; f1)
- 94,533BitextMining
- MTEB Tatoeba (epo-eng) (epo-eng; test; f1)
- 98,483BitextMining
- MTEB Tatoeba (kzj-eng) (kzj-eng; test; f1)
- 14,979BitextMining
- MTEB Tatoeba (awa-eng) (awa-eng; test; f1)
- 91,991BitextMining
- MTEB Tatoeba (fao-eng) (fao-eng; test; f1)
- 86,768BitextMining
- MTEB Tatoeba (mal-eng) (mal-eng; test; f1)
- 98,933BitextMining
- MTEB Tatoeba (ile-eng) (ile-eng; test; f1)
- 89,780BitextMining
- MTEB Tatoeba (bos-eng) (bos-eng; test; f1)
- 95,857BitextMining
- MTEB Tatoeba (cor-eng) (cor-eng; test; f1)
- 11,821BitextMining
- MTEB Tatoeba (cat-eng) (cat-eng; test; f1)
- 94,933BitextMining
- MTEB Tatoeba (eus-eng) (eus-eng; test; f1)
- 84,723BitextMining
- MTEB Tatoeba (yue-eng) (yue-eng; test; f1)
- 93,473BitextMining
- MTEB Tatoeba (swe-eng) (swe-eng; test; f1)
- 95,717BitextMining
- MTEB Tatoeba (dtp-eng) (dtp-eng; test; f1)
- 14,511BitextMining
- MTEB Tatoeba (kat-eng) (kat-eng; test; f1)
- 92,466BitextMining
- MTEB Tatoeba (jpn-eng) (jpn-eng; test; f1)
- 96,117BitextMining
- MTEB Tatoeba (csb-eng) (csb-eng; test; f1)
- 66,589BitextMining
- MTEB Tatoeba (xho-eng) (xho-eng; test; f1)
- 88,263BitextMining
- MTEB Tatoeba (orv-eng) (orv-eng; test; f1)
- 58,288BitextMining
- MTEB Tatoeba (ind-eng) (ind-eng; test; f1)
- 94,417BitextMining
- MTEB Tatoeba (tuk-eng) (tuk-eng; test; f1)
- 49,640BitextMining
- MTEB Tatoeba (max-eng) (max-eng; test; f1)
- 73,442BitextMining
- MTEB Tatoeba (swh-eng) (swh-eng; test; f1)
- 80,915BitextMining
- MTEB Tatoeba (hin-eng) (hin-eng; test; f1)
- 97,6BitextMining
- MTEB Tatoeba (dsb-eng) (dsb-eng; test; f1)
- 72,004BitextMining
- MTEB Tatoeba (ber-eng) (ber-eng; test; f1)
- 55,577BitextMining
- MTEB Tatoeba (tam-eng) (tam-eng; test; f1)
- 90,647BitextMining
- MTEB Tatoeba (slk-eng) (slk-eng; test; f1)
- 95,133BitextMining
- MTEB Tatoeba (tgl-eng) (tgl-eng; test; f1)
- 95,850BitextMining
- MTEB Tatoeba (ast-eng) (ast-eng; test; f1)
- 89,764BitextMining
- MTEB Tatoeba (mkd-eng) (mkd-eng; test; f1)
- 92,49BitextMining
- MTEB Tatoeba (khm-eng) (khm-eng; test; f1)
- 73,274BitextMining
- MTEB Tatoeba (ces-eng) (ces-eng; test; f1)
- 96,567BitextMining
- MTEB Tatoeba (tzl-eng) (tzl-eng; test; f1)
- 61,092BitextMining
- MTEB Tatoeba (urd-eng) (urd-eng; test; f1)
- 93,412BitextMining
- MTEB Tatoeba (ara-eng) (ara-eng; test; f1)
- 91,100BitextMining
- MTEB Tatoeba (kor-eng) (kor-eng; test; f1)
- 91,973BitextMining
- MTEB Tatoeba (yid-eng) (yid-eng; test; f1)
- 90,380BitextMining
- MTEB Tatoeba (fin-eng) (fin-eng; test; f1)
- 96,917BitextMining
- MTEB Tatoeba (tha-eng) (tha-eng; test; f1)
- 96,715BitextMining
- MTEB Tatoeba (wuu-eng) (wuu-eng; test; f1)
- 92,340BitextMining
- MTEB Touche2020 (test; ndcg_at_10)
- 27,249Retrieval
- MTEB ToxicConversationsClassification (test; accuracy)
- 71,059Classification
- MTEB TweetSentimentExtractionClassification (test; accuracy)
- 64,618Classification
- MTEB TwentyNewsgroupsClustering (test; v_measure)
- 51,329Clustering
- MTEB TwitterSemEval2015 (test; cos_sim_ap)
- 80,267PairClassification
- MTEB TwitterSemEval2015 (test; max_ap)
- 80,267PairClassification
- MTEB TwitterURLCorpus (test; cos_sim_ap)
- 87,117PairClassification
- MTEB TwitterURLCorpus (test; max_ap)
- 87,117PairClassification
Valores como impressos na ficha, com a configuração do próprio publicador. Não comparáveis entre publicadores e fora de qualquer ranking. Ficha do modelo
Limites e datas
O benchmark de qualidade publicado não testa uma versão local quantizada. Estimativas de memória não medem velocidade nem garantem compatibilidade. Tamanho de repositório, quando disponível, não é VRAM. Fórmulas
Página atualizada em · Catálogo revisado em · MTEB (Multilingual, v2)