CE=-∑y log pGQAKV cacheW+ΔWAx=bbeam=4embedding Ed_headPagedAttentiontr(A)decodeE[log pθ]reward rθPPL=exp(CE)log pRoPEMoEsoftmax(QKᵀ/√d)VEOSBayesn_headsblock tablehₜbatch BVar(X)advantage AₜKL(p||q)FFNθ ← θ-η∇Largmin L(θ)softmaxBOScos θtop-klim x→∞zᵢseq TGELUγaₙMLP∇θ Lrouter logitsWq≥sin θtop-pcausal mask
博客
文章千古事，得失寸心知。