HanseLM 78M Base - German-First Causal Language Model

HanseLM-78M-Base

HanseLM 78M Base is a German-first causal language model and experimental research preview trained completely from scratch. It was built to study whether a hybrid attention/convolution architecture can learn rich German text representations under a ~1.5-billion-token compute budget.

Hugging Face Model: Evicka/HanseLM-78M-Base

Model Specifications

Property	Value
Model Type	Decoder-only causal language model (`HanseForCausalLM`)
Parameters	78,430,848 (78M)
Vocabulary	24,576-token byte-level BPE with NFC normalization
Context Length	2,048 tokens
Hidden Size	640
Layers	14 total (11 attention blocks, 3 causal convolution blocks)
Attention	Grouped-Query Attention (GQA): 10 query heads, 2 KV heads (head dim 64)
Feed-Forward	SwiGLU (intermediate hidden size 1,792)
Position Encoding	Rotary Position Embeddings (RoPE, $\theta = 10{,}000$)
Normalization	Pre-norm RMSNorm ($\epsilon = 10^{-6}$) with Q/K per-head RMSNorm
Precision	BF16 training/autocast, FP32 master weights
Weight Tying	Input and output embeddings tied

Hybrid Architecture

Each block applies a pre-normalized token mixer and a pre-normalized SwiGLU feed-forward network with residual connections across 14 layers:

A  A  C  A  A  A  C  A  A  A  C  A  A  A

A (Attention Blocks): Causal scaled-dot-product Grouped-Query Attention. 10 query heads share 2 KV heads in 5-head groups with per-head RMS normalization prior to RoPE.
C (Convolution Blocks): Gated, depthwise causal 1D convolution with kernel size 7 at layers 3, 7, and 11.

Custom Tokenizer

Trained alongside the model with custom special tokens for chat/tooling scaffolding:

<|pad|> <|bos|> <|eos|> <|system|> <|user|> <|assistant|> <|tool|> <|tool_result|>

Pretraining Datasets

Trained on a curated mix of German and English web corpora:

HuggingFaceFW/fineweb-2
HuggingFaceFW/finewiki
HuggingFaceFW/fineweb

Quick Start via Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Evicka/HanseLM-78M-Base"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto"
)

prompt = "Die Hanse war"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50, do_sample=True, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

License

Model weights and implementation code are released under the MIT License.