Web crawler for creating personal copies of Japanese dictionaries
Go to file
2023-05-01 20:03:03 -05:00
bot Specify UTF-8 encoding with file i/o 2023-05-01 20:03:03 -05:00
data Add support for Shinmeikai 8th edition & Daijirin 4th edition 2023-05-01 17:31:28 -05:00
.gitignore First version 2023-04-07 22:05:36 -05:00
jitenbot.py Update jitenbot.py 2023-05-01 18:25:42 -05:00
LICENSE Initial commit 2023-04-07 16:37:51 -05:00
README.md Update README.md 2023-05-01 19:17:26 -05:00
requirements.txt Add support for Shinmeikai 8th edition & Daijirin 4th edition 2023-05-01 17:31:28 -05:00

jitenbot

Jitenbot is a program for scraping Japanese dictionary websites and compiling the scraped data into compact dictionary file formats.

Supported Dictionaries

Supported Output Formats

Usage

usage: jitenbot [-h] [-p PAGE_DIR] [-i IMAGE_DIR]
                {jitenon-yoji,jitenon-kotowaza,smk8,daijirin2}

Convert Japanese dictionary files to new formats.

positional arguments:
  {jitenon-yoji,jitenon-kotowaza,smk8,daijirin2}
                        name of dictionary to convert

options:
  -h, --help            show this help message and exit
  -p PAGE_DIR, --page-dir PAGE_DIR
                        path to directory containing XML page files
  -i IMAGE_DIR, --image-dir IMAGE_DIR
                        path to directory containing image folders (gaiji,
                        graphics, etc.)

Online Targets

Jitenbot will scrape the target website and save the pages to the user cache directory. As a courtesy to the website owners, jitenbot is configured to pause for 10 seconds between each page request. Consequently, a complete crawl of a target website may take several hours.

HTTP request headers (user agent string, etc.) may be customized by editing the config.json file created in the user config directory.

Offline Targets

Page data and image data must be procured by the user and passed to jitenbot via the appropriate command line flags.

Attribution

Adobe-Japan1_sequences.txt is provided by The Adobe-Japan1-7 Character Collection.

Examples

四字熟語辞典オンライン

白玉微瑕 (web)

yoji_hakugyokunobika_web

白玉微瑕 (yomichan)

yoji_hakugyokunobika


故事・ことわざ・慣用句オンライン

怒髪、冠を衝く (web)

kotowaza_dohatsu_web

怒髪、冠を衝く (yomichan)

kotowaza_dohatsu_yomi


新明解国語辞典 第八版

離れる (print)

smk8_hanareru_print

離れる (yomichan)

smk8_hanareru_yomichan


大辞林 第四版

令月 (print)

daijirin_reigetsu_print

令月 (yomichan)

daijirin_reigetsu_yomichan