Uniquer, v1.5

Utility for an increase of an uniqueness of a text.

Filtering of synonyms by their form via pymorphy2 library, an extension of a preparation of a synonyms dictionary.

Change Log

• make a dependency on pymorphy2 library optional;
• increase an uniqueness of a text:
  • support a filtering of synonyms by a word form (optionally);
• utility for a preparation of a synonyms dictionary:
  • extend a dictionary (optionally);
• prepare the synonyms dictionary.

Features

• increase an uniqueness of a text:
  • by a replacing of words to their synonyms via a synonyms dictionary:
    • load a dictionary to a prefix tree (a trie);
    • find a word (one to choose):
      • by its longest prefix;
      • by an exact match;
    • filter synonyms by a word form (a gender; via pymorphy2 library; optionally);
    • select a synonym randomly;
    • correct a synonym form (a case and a number) in accordance with a word form (via pymorphy2 library; optionally, but may be required);
  • by a double translation through an interim language (source → interim → source) via Google Translate service (optionally);
• support a simple direct translation from a specified language (without a replacing to synonyms);
• support restrictions of a replacing to synonyms:
  • minimal length of a replaced word;
  • minimal length of a selected synonym;
  • probability of a replacing of each word;
  • replace only nouns (detect a part of speech via pymorphy2 library; optionally);
• utility for a preparation of a synonyms dictionary:
  • filter words:
    • save only nouns (detect a part of speech via pymorphy2 library);
    • skip words from an exceptions dictionary (match words by their normal form via pymorphy2 library);
    • skip words with a small usage frequency according to a frequencies dictionary (match words by their normal form via pymorphy2 library):
      • filter by a minimal frequency (instances per million words);
      • filter by a minimal Juilland's D coefficient;
      • optionally require a checking of a frequency (by default words with an unknown frequency has been keep);
  • transform words to their normal form (via pymorphy2 library);
  • remove duplicates:
    • combine synonyms lists for same words;
    • remove duplicates from synonyms lists;
  • extend a dictionary (optionally): add each synonym of a word as a single word;
• correspond to constraints of Google Translate service:
  • split the text to chunks:
    • don't cut a sentence;
    • don't cut a word;
  • add a delay between requests;
• CLI:
  • input the text from stdin;
  • output the text to stdout;
• format an output (optionally):
  • extract a cut (first few sentences);
  • split a text to paragraphs.

Screenshots

Work without pymorphy2 library