计划继续在奇怪的地方重新启动

Question

更新：

所以问题似乎出在生成器上，而不一定与下一个令牌和前瞻函数有关。我在 set!s 发生的地方添加了一些显示调用，发现问题是在第二次调用 (generate-token) 后，它从第一次调用的位置恢复执行。

这是该程序的完整代码（我保留了下面的原始帖子以供参考）：

(define char-alphanumeric? (lambda (char) (or (char-alphabetic? char) (char-numeric? char))))

(define generate-token #f)
(define filename "input.txt")

(define next-token #f)
(define lookahead #f)
(define status #f)

(let ((f (open-input-file filename)) (yield #f) (token "") (lookahead-token #f) (current-token #f))
  (set! generate-token (lambda ()
                     (letrec ((next-char (lambda (c)
                                            (let ((separators (list #\; #\,)))
                                              (cond ((eof-object? c) (display "last token before eof: ") (display token) (newline) (yield c))
                                                    ((member c separators)
                                                     (begin
                                                            (display "token before sep: ") (display token) (newline)
                                                            (call-with-current-continuation (lambda (resume)
                                                                                              (set! generate-token (lambda () (resume)))
                                                                                              (yield token)))
                                                            (display "back from call") (set! token "")
                                                            (call-with-current-continuation (lambda (resume)
                                                                                              (set! generate-token (lambda () (resume)))
                                                                                              (yield (make-string 1 c))))
                                                            ))
                                                    ((or (char-alphanumeric? c) (equal? c #\_)) ; c is part of a string token
                                                     (begin (display "found char: ") (display c) (display "; added to string: ")
                                                            (set! token (string-append token (make-string 1 c)))
                                                            (display token) (newline)
                                                            (next-char (read-char f))))
                                                    ((char-whitespace? c)
                                                     (begin
                                                            (display "token before ws: ") (display token) (newline)
                                                            (if (> (string-length token) 0)
                                                                (begin (call-with-current-continuation (lambda (resume)
                                                                                              (display "setting generate-token to resume") (newline)
                                                                                              (set! generate-token (lambda ()
                                                                                                                 ((display "calling resume") (newline)
                                                                                                                 (resume))))
                                                                                              (display "yielding token from cc") (newline)
                                                                                              (yield token)))
                                                                 (display "continuing...") (newline)
                                                                (set! token ""))
                                                                ;(set! token "")
                                                                ))))
                                              (next-char (read-char f))
                                              ))))
                       (call-with-current-continuation (lambda (k) ((set! yield k) (k (next-char (read-char f))))))
                       )))
  (set! lookahead (lambda () (begin
                              (if (not lookahead-token)
                                  (begin (display "no lookahead") (newline)
                                         (display "setting lookahead-token") (newline)
                                         (set! lookahead-token (string-copy (generate-token)))
                                         (display "lookahead set to ") (display lookahead-token) (newline)
                                         ))
                             lookahead-token)))
  (set! next-token (lambda () (begin
                               (if lookahead-token
                                  (begin (display "affirmative") (newline)
                                         (set! current-token (string-copy lookahead-token))
                                         (set! lookahead-token #f))
                                  (begin (display "negative") (newline)
                                         (display "setting current token to next-token") (newline)
                                         (set! current-token (string-copy (generate-token)))
                                         (display "current token = ") (display current-token) (newline)
                                         (set! lookahead-token #f)))
                               current-token)))
  (set! status (lambda () (begin (display current-token) (display " -> ") (display lookahead-token) (newline))))
)

按照下面原始帖子中的第一个示例执行下一个令牌和前瞻调用会产生：

> (next-token)
negative
setting current token to next-token
found char: t; added to string: t
found char: h; added to string: th
found char: e; added to string: the
found char: s; added to string: thes
found char: e; added to string: these
token before ws: these
setting generate-token to resume
yielding token from cc
current token = these
"these"
> (status)
these -> #f
> (lookahead)
no lookahead
setting lookahead-token
calling resume
continuing...
found char: a; added to string: a
found char: r; added to string: ar
found char: e; added to string: are
token before ws: are
setting generate-token to resume
yielding token from cc ; the problem is right here: the generate token call is
current token = are    ; sending control back to next-token instead of lookahead.
"are"
> (status)
are -> #f

我不知道为什么会这样这样做，但会承认我是延续性的新手，并且可能不完全理解其后果。一如既往，任何帮助将不胜感激。

谢谢。

原始帖子如下：

---

我创建了一个生成器，它解析文本文件并一次返回一个标记作为字符串。因此，如果我有一个文件包含

these are my file contents

对（生成令牌）的连续调用，则分别返回“这些”“是”“我的”...。这似乎有效，但我将其编写为解析器的一部分以进行更大的作业。生成器似乎工作顺利，但当我构建 LR(1) 解析器来解析标记流时，我需要能够执行前瞻。为此，我创建了以下程序：

(define generate-token #f)
(define next-token #f)
(define lookahead #f)
(define status #f)

(let ((lookahead-token #f) (current-token #f))
  (set! generate-token (lambda () ... ) ; the generator function
  (set! lookahead (lambda () (begin
                              (if (not lookahead-token)
                                  (begin (display "no lookahead") (newline)
                                   (set! lookahead-token (string-copy (generate-token)))))
                             lookahead-token)))
  (set! next-token (lambda () (begin
                               (if lookahead-token
                                  (begin (display "affirmative") (newline)
                                         (set! current-token (string-copy lookahead-token))
                                         (set! lookahead-token #f))
                                  (begin (display "negative") (newline)
                                         (set! current-token (string-copy (generate-token)))
                                         (set! lookahead-token #f)))
                                  current-token)))
  (set! status (lambda () (begin (display current-token) (display " -> ") (display lookahead-token) (newline))))
)

但是，这些程序无法按预期工作。我的印象是方案（这是用 drRacket 编写的，但使用 #lang r5rs）按值传递对象，因此（字符串复制调用假设是不必要的，但这仍然无法按预期工作。它的工作方式如下：

> (status)
#f -> #f
> (next-token)
"these"
> (status) ; next-token properly sets current-token 
"these" -> #f
> (lookahead) ; generator returns "are" as expected
"are"
> (status) ; notice that the current-token has been replaced instead of the lookahead-token 
"are" -> #f

在不同的流程中，如果首先调用 (lookahead)，它就会正常工作。

> (status)
#f -> #f
> (lookahead)
"these"
> (status)
#f -> "these"
> (lookahead)
"these"
> (status)
#f -> "these"
> (next-token)
"these"
> (status)
"these" -> #f
> (lookahead)
"are"
> (status)
"these" -> "are"

如果有人知道发生了什么，任何见解都将不胜感激。披露：这是学校作业，但我不是在问你。为我做这件事>.>。

Answer 1

问题在于您的 yield 实现。为了便于阅读，以下是实现的简化版本：

(define yield #f)
(define my-generator
        (lambda ()
           (let forever ((num 0))
               (let/cc resume
                  (set! my-generator resume)
                  (yield num))
               (forever (add1 num)))))

(define zero (let/cc my-yield
                (set! yield my-yield)
                (my-generator)))
(define one (my-generator))
zero
;; => 1

延续总是跳回其返回值最初所在的位置。因此，如果您使用 call/cc 或 let/cc 捕获 yield，那么每次您 yield 时，您都会跳回到你捕获它的地方。

在您的代码中，您在每次 yield 之前都花时间更新了简历延续，但您只在您所在的一个位置更新了 yield
调用next-token。每次调用 generate-token 时，您都需要更新 yield。

对于那些不在学校这样做的读者，只需使用racket/generator，其中所有的工作和更多工作都会为您完成。