如何使用bertmodel读取Bertmumaskedlm？

发布于 2025-01-24 10:20:56 字数 487 浏览 2 评论 0原文

我有微调的bertformaskedlm，现在我想用bertmodel阅读它。但是我保存的模型看起来像这样：

BertForMaskedLM(
  (bert): BertModel(
    (embeddings): BertEmbeddings(
      (word_embeddings): Embedding(119547, 768, padding_idx=0)
      (position_embeddings): Embedding(512, 768)
      (token_type_embeddings): Embedding(2, 768)
      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
      (dropout): Dropout(p=0.1, inplace=False)
..................

如何使用变形金刚Bertmodel阅读它？

原文

I have fine-tuned BertForMaskedLM and now I want to read it with BertModel. But my saved model looks like this:

BertForMaskedLM(
  (bert): BertModel(
    (embeddings): BertEmbeddings(
      (word_embeddings): Embedding(119547, 768, padding_idx=0)
      (position_embeddings): Embedding(512, 768)
      (token_type_embeddings): Embedding(2, 768)
      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
      (dropout): Dropout(p=0.1, inplace=False)
..................

How to read it with transformers BertModel?

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

海螺姑娘 2025-01-31 10:20:56

我也遇到了同样的问题，我试图提供解决方案。首先，我必须确保以以下形式保存我的bertformaskedlm：

self.model.save_pretrained('bert_mlm_source')

其次，加载训练有素的模型时，我们只需要将bert layer的参数从bertformaskedlm中取出，因为

self.bert = BertModel.from_pretrained(self.cfg.bert_source)
self.mlm_bert = BertForMaskedLM.from_pretrained(self.cfg.bert_mlm_source)
self.bert.embeddings = self.mlm_bert.bert.embeddings
self.bert.encoder = self.mlm_bert.bert.encoder

我只需要使用自我即可。 bert'用于适当的下游任务。可以这样做，因为bertformaskedlm简单地在Bert的最后一个添加了Bertonlymlmhead层，而中间的密钥层是相同的。

如果您找到了它，可能会有一种更方便和美丽的方式，请告诉我。非常感谢。

I also encountered the same problem, I tried to give a solution. First, I must ensure that my BertForMaskedLM is saved in the following form:

self.model.save_pretrained('bert_mlm_source')

Second, when loading the trained model, we just need to take the parameters of Bert layer out of BertForMaskedLM as:

self.bert = BertModel.from_pretrained(self.cfg.bert_source)
self.mlm_bert = BertForMaskedLM.from_pretrained(self.cfg.bert_mlm_source)
self.bert.embeddings = self.mlm_bert.bert.embeddings
self.bert.encoder = self.mlm_bert.bert.encoder

In the end, I just need to use 'self.bert' for the appropriate downstream tasks. This can be done because BertForMaskedLM simply adds a layer of BertOnlyMLMHead on last of Bert, and the key layers in the middle are the same.

There may be a more convenient and beautiful way, if you have found it, please let me know. Thank you very much.

回复收藏 0 原文

~没有更多了~