References:
ar-en
Use the following command to load this dataset in TFDS:
ds = tfds.load('huggingface:opus_wikipedia/ar-en')
- Description:
This is a corpus of parallel sentences extracted from Wikipedia by Krzysztof Wołk and Krzysztof Marasek. Please cite the following publication if you use the data: Krzysztof Wołk and Krzysztof Marasek: Building Subject-aligned Comparable Corpora and Mining it for Truly Parallel Sentence Pairs., Procedia Technology, 18, Elsevier, p.126-132, 2014
20 languages, 36 bitexts
total number of files: 114
total number of tokens: 610.13M
total number of sentence fragments: 25.90M
- License: No known license
- Version: 1.0.0
- Splits:
Split | Examples |
---|---|
'train' |
151136 |
- Features:
{
"id": {
"dtype": "string",
"id": null,
"_type": "Value"
},
"translation": {
"languages": [
"ar",
"en"
],
"id": null,
"_type": "Translation"
}
}
ar-pl
Use the following command to load this dataset in TFDS:
ds = tfds.load('huggingface:opus_wikipedia/ar-pl')
- Description:
This is a corpus of parallel sentences extracted from Wikipedia by Krzysztof Wołk and Krzysztof Marasek. Please cite the following publication if you use the data: Krzysztof Wołk and Krzysztof Marasek: Building Subject-aligned Comparable Corpora and Mining it for Truly Parallel Sentence Pairs., Procedia Technology, 18, Elsevier, p.126-132, 2014
20 languages, 36 bitexts
total number of files: 114
total number of tokens: 610.13M
total number of sentence fragments: 25.90M
- License: No known license
- Version: 1.0.0
- Splits:
Split | Examples |
---|---|
'train' |
823715 |
- Features:
{
"id": {
"dtype": "string",
"id": null,
"_type": "Value"
},
"translation": {
"languages": [
"ar",
"pl"
],
"id": null,
"_type": "Translation"
}
}
en-sl
Use the following command to load this dataset in TFDS:
ds = tfds.load('huggingface:opus_wikipedia/en-sl')
- Description:
This is a corpus of parallel sentences extracted from Wikipedia by Krzysztof Wołk and Krzysztof Marasek. Please cite the following publication if you use the data: Krzysztof Wołk and Krzysztof Marasek: Building Subject-aligned Comparable Corpora and Mining it for Truly Parallel Sentence Pairs., Procedia Technology, 18, Elsevier, p.126-132, 2014
20 languages, 36 bitexts
total number of files: 114
total number of tokens: 610.13M
total number of sentence fragments: 25.90M
- License: No known license
- Version: 1.0.0
- Splits:
Split | Examples |
---|---|
'train' |
140124 |
- Features:
{
"id": {
"dtype": "string",
"id": null,
"_type": "Value"
},
"translation": {
"languages": [
"en",
"sl"
],
"id": null,
"_type": "Translation"
}
}
en-ru
Use the following command to load this dataset in TFDS:
ds = tfds.load('huggingface:opus_wikipedia/en-ru')
- Description:
This is a corpus of parallel sentences extracted from Wikipedia by Krzysztof Wołk and Krzysztof Marasek. Please cite the following publication if you use the data: Krzysztof Wołk and Krzysztof Marasek: Building Subject-aligned Comparable Corpora and Mining it for Truly Parallel Sentence Pairs., Procedia Technology, 18, Elsevier, p.126-132, 2014
20 languages, 36 bitexts
total number of files: 114
total number of tokens: 610.13M
total number of sentence fragments: 25.90M
- License: No known license
- Version: 1.0.0
- Splits:
Split | Examples |
---|---|
'train' |
572717 |
- Features:
{
"id": {
"dtype": "string",
"id": null,
"_type": "Value"
},
"translation": {
"languages": [
"en",
"ru"
],
"id": null,
"_type": "Translation"
}
}
en-vi
Use the following command to load this dataset in TFDS:
ds = tfds.load('huggingface:opus_wikipedia/en-vi')
- Description:
This is a corpus of parallel sentences extracted from Wikipedia by Krzysztof Wołk and Krzysztof Marasek. Please cite the following publication if you use the data: Krzysztof Wołk and Krzysztof Marasek: Building Subject-aligned Comparable Corpora and Mining it for Truly Parallel Sentence Pairs., Procedia Technology, 18, Elsevier, p.126-132, 2014
20 languages, 36 bitexts
total number of files: 114
total number of tokens: 610.13M
total number of sentence fragments: 25.90M
- License: No known license
- Version: 1.0.0
- Splits:
Split | Examples |
---|---|
'train' |
58116 |
- Features:
{
"id": {
"dtype": "string",
"id": null,
"_type": "Value"
},
"translation": {
"languages": [
"en",
"vi"
],
"id": null,
"_type": "Translation"
}
}