org.apache.lucene.analysis.cn.smart.hhmm

Class SegTokenFilter

java.lang.Object

org.apache.lucene.analysis.cn.smart.hhmm.SegTokenFilter

public class SegTokenFilter
extends Object

Filters a SegToken by converting full-width latin to half-width, then lowercasing latin. Additionally, all punctuation is converted into Utility.COMMON_DELIMITER

WARNING: This API is experimental and might change in incompatible ways in the next release.

Constructor Summary

Constructors

Constructor and Description
SegTokenFilter()

Method Summary

Modifier and Type	Method and Description
SegToken	filter(SegToken token) Filter an input SegToken

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SegTokenFilter

public SegTokenFilter()

Method Detail

filter

public SegToken filter(SegToken token)

Filter an input SegToken

Full-width latin will be converted to half-width, then all latin will be lowercased. All punctuation is converted into Utility.COMMON_DELIMITER

Parameters:

token - input SegToken

Returns:

normalized SegToken