Class Reference
IRIS for UNIX 2024.1.2
|
Class Reference
IRIS for UNIX 2024.1.2
|
|
|
Private
Storage
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This class contains utility methods to manage the word list used by the decompounding algorithm. Decompounding is about identifying the words making up a compound term, such as the words "thunder" and "storm" in the compound term "thunderstorms". It is used primarily for search purposes, allowing you to find records containing compounds of the search terms too. Lanugages like German, where compounding happens often, require decompounding support for a good search experience.
The decompounding algorithm supplied here requires a list of candidate words it will try to recognize in to-be-decompounded terms. These candidate words can be added through training the algorithm using any of the following methods, which accept free text that will be cut into candidate terms and then stripped of any recognizable compounds:
Alternatively, individual words can be added and removed through the
Decompounding is used by iFind indices who have their INDEXOPTION set to 2 (see also
The algorithm can also be invoked directly through a
// simple training
do ##class(%iKnow.Stemming.DecompoundUtils).AddWord("en", "thunder")
do ##class(%iKnow.Stemming.DecompoundUtils).AddWord("en", "storm")
// invoke decompounder
write ##class(%iKnow.Stemmer).GetDefault("en", .tStemmer)
write tStemmer.Decompound("thunderstorms", .tWords)
zwrite tWords
|
|
|
|
| Methods | |||
|---|---|---|---|
| AddWord | AppendTrainingDataFromDomain | AppendTrainingDataFromFiles | AppendTrainingDataFromQuery |
| CleanWordList | ClearTrainingData | NeverSeparate | RemoveWord |
|
|
Adds a word to the compound dictionary for the supplied language. The supplied word will be treated as a valid compound element the algorithm will no longer try to split in smaller elements. Optionally supply a positive integer frequency value to increase its weight when multiple options are available.
If pWord is also present in the list of strings never to split off through a call to
NeverSeparate , it will be removed from that list.When performing a lot of manual updates, it is recommended to set pClean=0 and only run the
CleanWords method once after all additions, to verify if these new additions indicate particular existing words should be removed as they turn out to be compounds themselves.
Appends word frequency information drawn from an existing iKnow domain to the word dictionary for decompounding in this namespace. When pEntType=$$$ENTTYPEANY (default), the full sentence values (with literal info) will be used to derive words. To restrict this to concepts or relations only, use $$$ENTTYPECONCEPT resp. $$$ENTTYPERELATION.
Multiple calls to this method (for different resultsets) will append to the existing info. Use
ClearTrainingData if you want to drop all existing data.When pClean=1, the generated word list will automatically be cleaned after loading the new data through a call to
CleanWordList . You may use pClean=0 and only callCleanWordList after appending training data from multiple sources, but it should be called once before decompounding any new words through the%iKnow.Stemmer object.
Appends word frequency information drawn from the *.txt files in pDirectory to the word dictionary for decompounding in this namespace. Multiple calls to this method (for different directories) will append to the existing info. Use
ClearTrainingData if you want to drop all existing data.When pClean=1, the generated word list will automatically be cleaned after loading the new data through a call to
CleanWordList . You may use pClean=0 and only callCleanWordList after appending training data from multiple sources, but it should be called once before decompounding any new words through the%iKnow.Stemmer object.
Appends word frequency information drawn from the first column of the supplied %ResultSet to the word dictionary for decompounding in this namespace. Multiple calls to this method (for different resultsets) will append to the existing info. Use
ClearTrainingData if you want to drop all existing data.When pClean=1, the generated word list will automatically be cleaned after loading the new data through a call to
CleanWordList . You may use pClean=0 and only callCleanWordList after appending training data from multiple sources, but it should be called once before decompounding any new words through the%iKnow.Stemmer object.
Clears any identifiable compounds from the current decompound dictionary for pLanguage. This method should be run at least once between appending data to the training set through any of the Append* methods in this class and using the Decompound() method in a%iKnow.Stemmer object.
Drops ALL training data for a given language. Use with care.
Marks pString as a character sequence that should never be split off and therefore never be returned as a compound element of its own. If this string was also part of the compound dictionary as a candidate, it will be removed automatically as if callingRemoveWord
Removes a word from the compound dictionary for the supplied language. This word will no longer be treated as a valid compound element. Use this to clear the list of eventual composite words added previously.