Analysis on the number of layers in the transformer-based model for neural machine translation

Dongxing Li; Zuying Luo

doi:10.1117/12.2661982

28 December 2022 Analysis on the number of layers in the transformer-based model for neural machine translation

Dongxing Li, Zuying Luo

Author Affiliations +

Proceedings Volume 12506, Third International Conference on Computer Science and Communication Technology (ICCSCT 2022); 1250625 (2022) https://doi.org/10.1117/12.2661982
Event: International Conference on Computer Science and Communication Technology (ICCSCT 2022), 2022, Beijing, China

Abstract

Recently, the transformer-based models have been widely used in sequence-to-sequence (seq2seq) tasks, especially neural machine translation (NMT). In the original transformer, the layer number in encoder is equal to the layer number in decoder. However, the structure is more complex and task is more difficult in decoder than those in encoder, so the layer number should not be same. In order to verify how many layer number in encoder and decoder is properly valued, we improve transformer as our model and conduct four experiments on four translation tasks of IWSLT2017. The experimental results show that the layer number in decoder should be larger than that in encoder, which can bring better translation performance.

Citation Download Citation

Dongxing Li and Zuying Luo "Analysis on the number of layers in the transformer-based model for neural machine translation", Proc. SPIE 12506, Third International Conference on Computer Science and Communication Technology (ICCSCT 2022), 1250625 (28 December 2022); https://doi.org/10.1117/12.2661982

Access the abstract

PROCEEDINGS
7 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Computer programming

Transformers

Analytical research

Data modeling

Head

Lithium

Matrices

Show All Keywords

RELATED CONTENT

Experimental analysis of position embedding in graph transformer networks
Proceedings of SPIE (May 12 2022)

Knowledge base question answering based on regularization and feature fusion
Proceedings of SPIE (October 07 2022)

Low rank adaptation with gating mechanisms in large language models,...
Proceedings of SPIE (October 16 2024)

BERT-based regression model for micro-edit humor classification task
Proceedings of SPIE (October 15 2021)

Chinese named entity recognition incorporating syntactic information
Proceedings of SPIE (April 28 2023)

Chinese classical poetry generation LoRA for T5
Proceedings of SPIE (October 19 2023)

Attentional SoftLexicon for Chinese named entity recognition
Proceedings of SPIE (August 16 2024)

Subscribe to Digital Library

Receive Erratum Email Alert

Show All Keywords

Keywords/Phrases

Search In:

Publication Years