Yearly Model Data

Word embedding statistics for each year of the Mahua corpus. Each year includes three model types: Word2Vec, FastText, and BERT.

1955 Model Statistics

Corpus: 4 files (Nov-Dec 1955)

Word2Vec

Vocabulary Size	3,674 words
Vector Size	100 dimensions
Top Words	來 (116), 馬來亞 (97), 生活 (88)
File	1955_model_data_word2vec.json

Download Data

FastText

Vocabulary Size	3,674 words
Vector Size	100 dimensions
Features	Subword information included
File	1955_model_data_fasttext.json

Download Data

BERT

Sentence Count	366 sentences
Embedding Size	768 dimensions
Word Count	8,424 words
File	1955_model_data_bert.json

Download Data

1956 Model Statistics

Corpus: 12 files (Jan-Dec 1956)

Word2Vec

Vocabulary Size	~5,000 words (estimated)
Vector Size	100 dimensions
File	1956_model_data_word2vec.json

Download Data

FastText

Vocabulary Size	~5,000 words
Vector Size	100 dimensions
File	1956_model_data_fasttext.json

Download Data

BERT

Sentence Count	~500 sentences (estimated)
Embedding Size	768 dimensions
File	1956_model_data_bert.json

Download Data

1957 Model Statistics

Corpus: 12 files (Jan-Dec 1957)

Word2Vec

Vocabulary Size	~5,500 words (estimated)
Vector Size	100 dimensions
File	1957_model_data_word2vec.json

Download Data

FastText

Vocabulary Size	~5,500 words
Vector Size	100 dimensions
File	1957_model_data_fasttext.json

Download Data

BERT

Sentence Count	~550 sentences
Embedding Size	768 dimensions
File	1957_model_data_bert.json

Download Data

1958 Model Statistics

Corpus: 12 files (Jan-Dec 1958)

Word2Vec

Vocabulary Size	~6,000 words (estimated)
Vector Size	100 dimensions
File	1958_model_data_word2vec.json

Download Data

FastText

Vocabulary Size	~6,000 words
Vector Size	100 dimensions
File	1958_model_data_fasttext.json

Download Data

BERT

Sentence Count	~600 sentences
Embedding Size	768 dimensions
File	1958_model_data_bert.json

Download Data

1959 Model Statistics

Corpus: 11 files (Feb-Dec 1959, excluding Jan)

Word2Vec

Vocabulary Size	~5,800 words (estimated)
Vector Size	100 dimensions
File	1959_model_data_word2vec.json

Download Data

FastText

Vocabulary Size	~5,800 words
Vector Size	100 dimensions
File	1959_model_data_fasttext.json

Download Data

BERT

Sentence Count	~580 sentences
Embedding Size	768 dimensions
File	1959_model_data_bert.json

Download Data

Special: Rationality Subcorpus (1959_04)

jf78 (April 1959, 1st issue) - Focused analysis on rationality concepts

Word2Vec (jf78)

Vocabulary Size	1,330 words
Vector Size	100 dimensions
File	1959_04_1_jf78_model_data_word2vec.json

Download Data

FastText (jf78)

Vocabulary Size	1,330 words
Vector Size	100 dimensions
File	1959_04_1_jf78_model_data_fasttext.json

Download Data

BERT (jf78)

Sentence Count	366 sentences
Embedding Size	768 dimensions
File	1959_04_1_jf78_model_data_bert.json

Download Data

1960 Model Statistics

Corpus: 12 files (Jan-Dec 1960)

Word2Vec

Vocabulary Size	~6,500 words (estimated)
Vector Size	100 dimensions
File	1960_model_data_word2vec.json

Download Data

FastText

Vocabulary Size	~6,500 words
Vector Size	100 dimensions
File	1960_model_data_fasttext.json

Download Data

BERT

Sentence Count	~650 sentences
Embedding Size	768 dimensions
File	1960_model_data_bert.json

Download Data

1961 Model Statistics

Corpus: 1 file (Jan 1961, 100th issue)

Word2Vec

Vocabulary Size	~1,500 words (estimated)
Vector Size	100 dimensions
File	1961_model_data_word2vec.json

Download Data

FastText

Vocabulary Size	~1,500 words
Vector Size	100 dimensions
File	1961_model_data_fasttext.json

Download Data

BERT

Sentence Count	~150 sentences
Embedding Size	768 dimensions
File	1961_model_data_bert.json

Download Data

Sample Data Preview

Sample Word2Vec vectors from 1959 data:

View Sample Vectors ▼

{
  "word2vec": {
    "model_info": {
      "vocabulary_size": 1330,
      "vector_size": 100,
      "vector_sample": {
        "來": [0.0235, -0.0242, 0.0451, -0.0123, ...],
        "馬來亞": [0.0452, -0.0312, 0.0289, 0.0567, ...],
        "生活": [0.0123, 0.0567, -0.0189, 0.0234, ...]
      },
      "similarity_sample": {
        "來": [
          ["馬來亞", 0.9872],
          ["文化", 0.9863],
          ["因為", 0.9851],
          ["時候", 0.9844]
        ]
      }
    }
  }
}

Note: Full vectors contain 100 dimensions per word. Download JSON files for complete data.

Download All Yearly Model Data

Get all model data files (Word2Vec, FastText, BERT) for every year in a single download.

View Download Options