M.H. Gafurov, Ibrohimi Yu

Annotation:

This article describes the process of analyzing and processing textual information by developing a private set of n- grams (unigrams, bigrams, and trigrams) from text elements. This method is of key importance in fields such as Natural Language Processing (NLP), linguistic modeling, and cryptography for the transformation of text objects. The article presents algorithms for dividing these elements, in which unigrams - each text character as a separate unit - while bigrams and trigrams are divided into groups of two or three characters, respectively. If there are not enough characters to create n-grams, a special "zero place" character (ø) is added to the end of the text object. The C# programming language is used for the practical implementation of these algorithms, and examples of its program code are provided. A significant portion of the work is devoted to the creation of sets of unique n-grams. It is noted that the best method for this is the use of "hash sets" (HashSet), since it is more efficient in terms of execution speed and ease of implementation.
Keywords: text, algorithm, unigram, bigram, trigram, private unique alphabet.