ModelCenter’s Documentation

ModelCenter implements PLMs (Pretrained Language Models) based on BMTrain backend.

Main Advantages:

• Low-Resource

• Efficient

• Extendable

GETTING STARTED

• Installation
  • 1. From PyPI (Recommend)
  • 2. From Source
• Quick start
  • Initialize bmtrain backend
  • Prepare the model
  • Perpare the dataset
  • Train the model
  • Run your code
• Benchmark
  • Comparison between Hugging Face Transformers
  • Comparison between Deepspeed ZeRO
• How to write a new model
  • Model Implementation
  • Model Config
• Pretrain data processing
  • 1. Raw data source
  • 2. Tokenize
  • 3. Self-Supervised Dataset

Models

• BERT
• GPT2
• GPT-j
• T5
• CPM1
• CPM2

PACKAGE REFERENCE

• module
  • Linear
  • Embedding
  • RelativePositionEmbedding
  • RotaryEmbedding
  • LayerNorm
  • Attention
  • FeedForward
• block
  • Encoder
  • Decoder
  • TransformerBlock
  • FFNBlock
  • SelfAttentionBlock
  • CrossAttentionBlock

Indices and tables

• Index