这个 Lucene TokenFilter 有什么问题吗？

Question

免责声明：过去 41 小时中，我有 36 个小时都在编码。我头疼。我不明白为什么这个组合 TokenFilter 返回 2 个令牌，都是来自源流的第一个令牌。

public class TokenCombiner extends TokenFilter {

  /*
   * Recombines all tokens back into a single token using the specified delimiter.
   */
  public TokenCombiner(TokenStream in, int delimiter) {
    super(in);
    this.delimiter = delimiter;
  }
  int delimiter;


  private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class);
  private final OffsetAttribute offsetAtt = addAttribute(OffsetAttribute.class);


  private boolean firstToken = true;
  int startOffset = 0;
  @Override
  public final boolean incrementToken() throws IOException {
    while (true){ 
        boolean eos = input.incrementToken(); //We have to process tokens even if they return end of file.
        CharTermAttribute token = input.getAttribute(CharTermAttribute.class);
        if (eos && token.length() == 0) break; //Break early to avoid extra whitespace.
        if (firstToken){
            startOffset = input.getAttribute(OffsetAttribute.class).startOffset();
            firstToken = false;

        }else{
            termAtt.append(Character.toString((char)delimiter));
        }
        termAtt.append(token);
        if (eos) break;
    }
    offsetAtt.setOffset(startOffset, input.getAttribute(OffsetAttribute.class).endOffset());
    return false;
  }

  @Override
  public void reset() throws IOException {
    super.reset();
    firstToken = true;
    startOffset = 0;
  }
}

Answer 1

我认为这里的根本问题是，您必须意识到 TokenCombiner 和它消耗（输入）的生产者共享并重用相同的属性！所以 token == termAtt 总是（尝试添加一个断言！）。

伙计，如果你周末编码了 36 个小时，那就太糟糕了……试试这个：

public class TokenCombiner extends TokenFilter {
  private final StringBuilder sb = new StringBuilder();
  private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class);
  private final OffsetAttribute offsetAtt = addAttribute(OffsetAttribute.class);
  private final char separator;
  private boolean consumed; // true if we already consumed

  protected TokenCombiner(TokenStream input, char separator) {
    super(input);
    this.separator = separator;
  }

  @Override
  public final boolean incrementToken() throws IOException {
    if (consumed) {
      return false; // don't call input.incrementToken() after it returns false
    }
    consumed = true;

    int startOffset = 0;
    int endOffset = 0;

    boolean found = false; // true if we actually consumed any tokens
    while (input.incrementToken()) {
      if (!found) {
        startOffset = offsetAtt.startOffset();
        found = true;
      }
      sb.append(termAtt);
      sb.append(separator);
      endOffset = offsetAtt.endOffset();
    }

    if (found) {
      assert sb.length() > 0; // always: because we append separator
      sb.setLength(sb.length() - 1);
      clearAttributes();
      termAtt.setEmpty().append(sb);
      offsetAtt.setOffset(startOffset, endOffset);
      return true;
    } else {
      return false;
    }
  }

  @Override
  public void reset() throws IOException {
    super.reset();
    sb.setLength(0);
    consumed = false;
  }
}