go>> scrape-goodreads>> 返回
项目作者： arthurdjn

项目描述 ：
Python package to scrape data from goodreads.com website. Authors, Books, and Quotes can be extracted.
高级语言： Python
项目地址: git://github.com/arthurdjn/scrape-goodreads.git
创建时间： 2020-05-21T16:05:25Z
项目社区：https://github.com/arthurdjn/scrape-goodreads
开源协议：MIT License
下载




scrape-goodreads

Python package to scrape data from goodreads.com website. Authors, Books, and Quotes can be extracted.

Project made during a Deep Learning project for poem generation using GPT2 model.

Installation

Install scrapereads package from PyPi:

  1. pip install scrapereads

Or from GitHub:

  1. git clone https://github.com/arthurdjn/scrape-goodreads
  2. cd scrape-goodreads
  3. pip install .

Getting Started

GoodReads API

You can search for Author, Book or Quote from the API:

  1. from scrapereads import GoodReads
  3. # Connect to the API
  4. goodreads = GoodReads()
  6. # Search for an author, from it's ID.
  7. AUTHOR_ID = 3389
  8. author = goodreads.search_author(AUTHOR_ID)
  9. # Search for a book
  10. BOOK_ID = 3048970
  11. book = goodreads.search_book(AUTHOR_ID, BOOK_ID)
  12. # Look for the 10 first books (set it to ``top_k=None`` to turn it off)
  13. books = goodreads.search_books(AUTHOR_ID, top_k=10)
  14. # ...Or quotes
  15. quotes = goodreads.search_quotes(AUTHOR_ID, top_k=5)

Quotes are made of a text, but optional information can be added (like number of likes, tags,
reference etc.)

  1. quotes = goodreads.search_quotes(AUTHOR_ID, top_k=5)
  3. for quote in quotes:
  4.     print(quote)
  5.     print()

Output:

  1. "Books are a uniquely portable magic."
  2. - Stephen King, from "On Writing: A Memoir Of The Craft"
  3.   Likes: 16225, Tags: books, magic, reading
  5. "If you don't have time to read, you don't have the time (or the tools) to write. Simple as that."
  6. - Stephen King
  7.   Likes: 12565, Tags: reading, writing
  9. "Get busy living or get busy dying."
  10. - Stephen King, from "Different Seasons"
  11.   Likes: 9014, Tags: life
  13. "Books are the perfect entertainment: no commercials, no batteries, hours of enjoyment for each dollar spent. What I wonder is why everybody doesn't carry a book around for those inevitable dead spots in life."
  14. - Stephen King
  15.   Likes: 8667, Tags: books
  17. "When his life was ruined, his family killed, his farm destroyed, Job knelt down on the ground and yelled up to the heavens, "Why god? Why me?" and the thundering voice of God answered, There's just something about you that pisses me off."
  18. - Stephen King, from "Storm Of The Century"
  19.   Likes: 7686, Tags: god, humor, religion

Structure

The package is divided as follows:

  • Author
  • Book, inherits from Author
  • Quote, inherits from Book

They share an abstract structure with AuthorMeta, BookMeta, QuoteMeta.

Retrieve data

Once you have one of these objects, you can also access data directly through their methods:

  1. author = goodreads.search_author(AUTHOR_ID)
  2. books = author.get_books()
  3. quotes = author.get_quotes()
  5. # Idem from an book
  6. book = goodreads.search_book(AUTHOR_ID, BOOK_ID)
  7. quotes = book.get_quotes()

In addition, you can retrieve parent objects from children:

  1. author = goodreads.search_author(AUTHOR_ID)
  2. quotes = author.get_quotes(top_k=10)
  3. quote = quotes[0]
  5. # Access to parent classes
  6. book = quote.get_book()
  7. author = quote.get_author()

You can scrape for description, links and other details information:

  1. author = goodreads.search_author(AUTHOR_ID)
  2. info = author.get_info()  # description of the author (genre, description, references etc.)

You can finally retrieve similar authors from an author with:

  1. author = goodreads.search_author(AUTHOR_ID)
  2. authors = author.get_similar_authors(top_k=5)

Save and export

You can save data in a JSON format (and encode it to ASCII if you want).

  1. author = goodreads.search_author(AUTHOR_ID)
  2. author_data = author.to_json(encode='ascii')
  3. # Idem for book and quote