Uniquer, v1.7
Posted on
Utility for an increase of an uniqueness of a text.
Save paragraphs on a working with rewrite services and censor a text by a replacing of words from an exceptions dictionary.
Change Log
- save paragraphs on a working with rewrite services (via an escaping of paragraphs separators);
- censor a text by a replacing of words from an exceptions dictionary (optionally):
- support a specification of an exceptions replacement;
- load a dictionary to a prefix tree (a trie);
- find an exception (all cases simultaneously):
- by an exact match;
- by its prefix:
- dynamically calculate a prefix length (as a specified percent of an exception length);
- restrict a prefix length by a specified minimum.
Features
- increase an uniqueness of a text:
- by a replacing of words to their synonyms via a synonyms dictionary (one to choose):
- by own algorithm:
- load a dictionary to a prefix tree (a trie);
- find a word (one to choose):
- by its longest prefix;
- by an exact match;
- filter synonyms by a word form (a gender; via pymorphy2 library; optionally);
- select a synonym randomly;
- correct a synonym form (a case and a number) in accordance with a word form (via pymorphy2 library; optionally, but may be required);
- via rewrite services (one to choose):
- by own algorithm:
- by a double translation through an interim language (source → interim → source) via Google Translate service (optionally);
- by a replacing of words to their synonyms via a synonyms dictionary (one to choose):
- support a simple direct translation from a specified language (without a replacing to synonyms);
- support restrictions of a replacing to synonyms:
- minimal length of a replaced word;
- minimal length of a selected synonym;
- probability of a replacing of each word;
- replace only nouns (detect a part of speech via pymorphy2 library; optionally);
- censor a text by a replacing of words from an exceptions dictionary (optionally):
- support a specification of an exceptions replacement;
- load a dictionary to a prefix tree (a trie);
- find an exception (all cases simultaneously):
- by an exact match;
- by its prefix:
- dynamically calculate a prefix length (as a specified percent of an exception length);
- restrict a prefix length by a specified minimum;
- utility for a preparation of a synonyms dictionary:
- filter words:
- save only nouns (detect a part of speech via pymorphy2 library);
- skip words from an exceptions dictionary (match words by their normal form via pymorphy2 library);
- skip words with a small usage frequency according to a frequencies dictionary (match words by their normal form via pymorphy2 library):
- filter by a minimal frequency (instances per million words);
- filter by a minimal Juilland's D coefficient;
- optionally require a checking of a frequency (by default words with an unknown frequency has been keep);
- transform words to their normal form (via pymorphy2 library);
- remove duplicates:
- combine synonyms lists for same words;
- remove duplicates from synonyms lists;
- extend a dictionary (optionally): add each synonym of a word as a single word;
- filter words:
- correspond to constraints of services:
- services:
- Google Translate service;
- rewrite services:
- constraints:
- split the text to chunks:
- don't cut a sentence;
- don't cut a word;
- add a delay between requests;
- split the text to chunks:
- services:
- CLI:
- input the text from stdin;
- output the text to stdout;
- format an output (optionally):
- extract a cut (first few sentences);
- split a text to paragraphs.