启动法语拼写纠正基于fastText的python脚本时,终端崩溃
试图
根据fastText
运行法语拼写校正的Python脚本(来自教程)。
我所做的工作
- 下载了
fastText
models (( bin and text) - 构建
fastText
作为命令行工具:
$ git clone https://github.com/facebookresearch/fastText.git
$ cd fastText
$ make
- 尝试运行下面的代码
script.py
import io
import fasttext
def load_vectors(fname):
fin = io.open(fname, 'r', encoding='utf-8', newline='\n', errors='ignore')
n, d = map(int, fin.readline().split())
data = {}
for line in fin:
tokens = line.rstrip().split(' ')
data[tokens[0]] = map(float, tokens[1:])
return data
def spelltest(tests, model, vocab):
"Run correction(wrong) on all (right, wrong) pairs; report results."
import time
start = time.clock()
good, unknown = 0, 0
n = len(tests)
for right, wrong in tests:
w = wrong
if w in vocab:
print('word: {} exists in the vocabulary. No correction required'.format(w))
else:
w_old = w
w = model.get_nearest_neighbors(w, k=1)[0][1]
print("found replacement: {} for word: {}".format(w, w_old))
good += (w == right)
dt = time.clock() - start
print('{:.0%} of {} correct at {:.0f} words per second '
.format(good / n, n, n / dt))
def Testset(lines):
"Parse 'right: wrong1 wrong2' lines into [('right', 'wrong1'), ('right', 'wrong2')] pairs."
return [(right, wrong)
for (right, wrongs) in (line.split(':') for line in lines)
for wrong in wrongs.split()]
if __name__ == "__main__":
model = fasttext.load_model("cc.fr.300.bin")
vocab = load_vectors("cc.fr.300.vec")
spelltest(Testset(open('Memoires_secrets_09.txt')), model, vocab)
#spelltest(Testset(open('spell-testset2.txt')), model, vocab)
错误
终端在运行时终端发出警告并崩溃script.py
。
Warning : `load_model` does not return WordVectorModel or SupervisedModel any more, but a `FastText` object which is very similar.
环境
Ubuntu 22.04 LTS
Python 3.10.4
Background
Trying to run the python script for French spelling correction based on fasttext
(from the tutorial here) from the command line.
What I have done
- downloaded the
fasttext
models (bin and text) - built
fasttext
as a command-line tool:
$ git clone https://github.com/facebookresearch/fastText.git
$ cd fastText
$ make
- tried to run the code below
Code
script.py
import io
import fasttext
def load_vectors(fname):
fin = io.open(fname, 'r', encoding='utf-8', newline='\n', errors='ignore')
n, d = map(int, fin.readline().split())
data = {}
for line in fin:
tokens = line.rstrip().split(' ')
data[tokens[0]] = map(float, tokens[1:])
return data
def spelltest(tests, model, vocab):
"Run correction(wrong) on all (right, wrong) pairs; report results."
import time
start = time.clock()
good, unknown = 0, 0
n = len(tests)
for right, wrong in tests:
w = wrong
if w in vocab:
print('word: {} exists in the vocabulary. No correction required'.format(w))
else:
w_old = w
w = model.get_nearest_neighbors(w, k=1)[0][1]
print("found replacement: {} for word: {}".format(w, w_old))
good += (w == right)
dt = time.clock() - start
print('{:.0%} of {} correct at {:.0f} words per second '
.format(good / n, n, n / dt))
def Testset(lines):
"Parse 'right: wrong1 wrong2' lines into [('right', 'wrong1'), ('right', 'wrong2')] pairs."
return [(right, wrong)
for (right, wrongs) in (line.split(':') for line in lines)
for wrong in wrongs.split()]
if __name__ == "__main__":
model = fasttext.load_model("cc.fr.300.bin")
vocab = load_vectors("cc.fr.300.vec")
spelltest(Testset(open('Memoires_secrets_09.txt')), model, vocab)
#spelltest(Testset(open('spell-testset2.txt')), model, vocab)
Error
The terminal throws a warning and crashes when running script.py
.
Warning : `load_model` does not return WordVectorModel or SupervisedModel any more, but a `FastText` object which is very similar.
Environment
Ubuntu 22.04 LTS
Python 3.10.4
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论