Machine Learning
TinyCoherent
An experimental pure C Transformer training and inference engine built for Apple Silicon. Explores 3.45M and 23M parameter architectures, subword BPE tokenization designed to address character-level sequence expansion, pure C conversational instruction fine-tuning with prompt-loss masking, pre-allocated activation buffer inspection via a live terminal REPL, and the Rung 6 V2 reasoning dataset curation workflow.
Research question
Can an inspectable glass-box Transformer language model be trained, instruction-tuned, and scaled entirely in pure C directly on Apple Silicon without dependency on Python/PyTorch runtimes, while maintaining rigorous reasoning token curation?
Method & architecture
TinyCoherent implements an autoregressive decoder-only Transformer entirely in ANSI C with zero third-party framework dependencies. The system architecture coordinates contiguous memory-mapped tensor buffers, custom analytical reverse-mode backpropagation, cache-aligned matrix multiplication routines via Apple Accelerate BLAS, causal self-attention, and an interactive glass-box REPL for inspecting internal activations in real-time.
- Zero Framework Overhead: Standalone ANSI C implementation without Python, PyTorch, CUDA, or external math framework dependencies (SOURCE-VERIFIED).
- Contiguous Activation Buffer Inspection: Layer activations and attention weights reside in pre-allocated memory buffers, enabling constant-time O(1) buffer lookup during generation without dynamic heap allocation per inspection tap (DESIGN CLAIM / HYPOTHESIS).
- Data Curation Integrity: Prioritizes verified step-by-step scratchpad examples over raw synthetic token volume, discarding unverified candidate tokens (AUTHOR-REPORTED).
- Hardware-Targeted Implementation: Integrates Apple Accelerate BLAS (cblas_sgemm) on unified memory architectures (SOURCE-VERIFIED; compute bandwidth bounds AUTHOR-REPORTED).
Results & findings
Character-Level Sequence Length Inflation
OBSERVEDCharacter-level tokenization expands multi-digit arithmetic and technical words into excessively long sequences, rapidly consuming available context capacity.
Apple Silicon BLAS Matrix Acceleration
SUPPORTEDcblas_sgemm BLAS integration enables zero-Python matrix multiplication on Apple Silicon; formal compute bandwidth benchmarks remain author-reported.
Prompt-Loss Masking in Pure C
SUPPORTEDZeroing loss gradients across prompt tokens isolates parameter updates to assistant responses during C conversational fine-tuning.
Token Curation vs Volume in Rung 6 V2
OBSERVEDDiscarding 1.5M unverified candidate tokens in favor of verified step-by-step scratchpads improved training signal quality according to author research logs.