Nagi-ovo
Follow
Breezing
110
Followers
10
Followings
Patron
主页
Archives
Portfolios
日寄
BPE
Latest
Hottest
Most Commented
LLM 演进史(六):揭开 Tokenizer 的神秘面纱
Tokenizer 是 LLM 中很重要但又没那么 fancy 的组件,在本系列之前的语言模型建模中,tokenizer 的实现方式是字符级的,将所有可能出现的 65 种字符制作嵌入表,然后用 embedding layer 对训练集进行编码向量化。而实践中…
LLM
15 min
2 months ago
Ownership of this blog data is guaranteed by blockchain and smart contracts to the creator alone.
Blockchain ID
#61009
Owner
0x6380302480224d53ec4c2c318d1c7be2c55a7582
Transaction Hash
Creation 0xe99aa0c3...eadd5b56dd
Last Update 0x87a7dc83...b5a81bfcb1
IPFS Address
ipfs://QmZhuq5TxuSMGBu1Q7F9Ki3vUTHL5ddikGqpNuaM2eCgyN