Uniquer, v1.3

Utility for an increase of an uniqueness of a text.

Filtering of a synonyms dictionary.

Change Log

• add the utility for a filtering of a synonyms dictionary:
  • save only nouns;
  • skip words from an exceptions dictionary;
  • skip words with a small usage frequency according to a frequencies dictionary:
    • filter by a minimal frequency;
    • filter by a minimal Juilland's D coefficient;
    • optionally require a checking of a frequency;
• prepare the exceptions dictionary;
• prepare the frequencies dictionary.

Features

• increase an uniqueness of a text:
  • by a replacing of words to their synonyms via a synonyms dictionary:
    • load a dictionary to a prefix tree (trie);
    • find a word by its longest prefix;
    • select a synonym randomly;
  • by a double translation through an interim language (source → interim → source) via Google Translate service;
• support a simple direct translation from a specified language (without a replacing to synonyms);
• prepare a synonyms dictionary on a loading:
  • combine synonyms lists for same words;
  • remove duplicates from synonyms lists;
• support restrictions of a replacing to synonyms:
  • minimal length of a replaced word;
  • minimal length of a selected synonym;
  • probability of a replacing of each word;
• utility for a filtering of a synonyms dictionary:
  • save only nouns (detect a part of speech via pymorphy2 library);
  • skip words from an exceptions dictionary (match words by their normal form via pymorphy2 library; see below for details);
  • skip words with a small usage frequency according to a frequencies dictionary (match words by their normal form via pymorphy2 library; see below for details):
    • filter by a minimal frequency (instances per million words);
    • filter by a minimal Juilland's D coefficient;
    • optionally require a checking of a frequency (by default words with an unknown frequency has been keep);
• correspond to constraints of Google Translate service:
  • split the text to chunks:
    • don't cut a sentence;
    • don't cut a word;
  • add a delay between requests;
• CLI:
  • input the text from stdin;
  • output the text to stdout;
• format an output (optionally):
  • extract a cut (first few sentences);
  • split a text to paragraphs.

Exceptions Dictionary

Exceptions dictionary format in the JSON Schema format:

{
  "type": "array",
  "minItems": 1,
  "uniqueItems": true,
  "items": {
    "type": "string",
    "minLength": 1
  }
}

Example:

[
  "дурак",
  "негодяй"
]

Frequencies Dictionary

Frequencies dictionary must be represented in the CSV format. It must have follow columns:

Number	Meaning	Type	Note
1	word	string
2	frequency (instances per million words)	real number	is greater than zero
3	Juilland's D coefficient	integer	belongs to the range [0; 100]

Header record is forbidden.

Example:

волшебник,10.1,76
рациональность,3.0,71