From-scratch Transformer · Machine Translation

श्रुतम्

ShrutamMT v1

"Śrutam" — that which is heard. A single Transformer, hand-built and trained from zero, that reads eleven Indic scripts and speaks back in English.

Eleven scripts, one bridge

Every source sentence carries a language tag — <hi>, <bn>, <ta> — so a single shared encoder learns all eleven at once.

অসমীয়া বাংলা ગુજરાતી हिन्दी ಕನ್ನಡ മലയാളം मराठी ଓଡ଼ିଆ ਪੰਜਾਬੀ தமிழ் తెలుగు → EN

Best result

Step 400,000 of 400,000 — checkpoint released as-is, no cherry-picking.

Overall BLEU
29.76
Overall chrF
52.61
Val perplexity
16.91
Val loss
2.83
Parameters
148.5M
Training pairs
49.2M

Training curve

Validation perplexity nearly halved between step 20,000 and step 400,000, once the Noam learning-rate schedule cleared its mid-training plateau.

Val loss Val perplexity (log scale)

Per-language ledger

30 held-out sentences per language, beam search (size 4). Quality tracks training-data volume closely — Bengali had 8.5M pairs, Assamese had 137K.

LanguageCodeBLEUchrF

How it reads

tokenizer

64K joint SentencePiece BPE spanning all eleven scripts plus English, with language tags reserved as single, unsplittable tokens.

encoder–decoder

6+6 layers, d_model 768, 12 heads — the original Vaswani et al. architecture, hand-written in PyTorch with no pretrained components.

tied weights

Source embedding, target embedding, and output projection share one weight matrix — fewer parameters, better cross-lingual generalization.

decoding

Beam search, width 4, length-penalty 0.6, run per-language at evaluation time for faithful BLEU/chrF scoring.