Web Scraping using BeautifulSoup library.

 What is it a Beautifulsoup library? Beautiful Soup is a Python library for pulling data out of HTML and XML files. It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree.

To install Beautifulsoup: pip install bs4

What is the requests library? The library is the de facto standard for making HTTP requests in Python. It abstracts the complexities of making requests behind a beautiful, simple API so that you can focus on interacting with services and consuming data in your application.

To install requests: pip install requests

What is the html5lib library? html5lib is a pure python library for parsing HTML. It is designed to conform to the WHATWG HTML specification, as is implemented by all major web browsers.  

To install html5lib: pip install html5lib

Steps:

  1. Installing the required third-party libraries.
  2. Accessing the HTML content from the webpage.
  3. Parsing the HTML content.
  4. Searching and navigating through the parse tree.




The BeautifulSoup library is that it is built on top of the HTML parsing libraries like html5lib, lxml, HTML. parser, etc.

soup = BeautifulSoup(r.content, 'html5lib')

We create a BeautifulSoup object by passing two arguments.
  1. r.content: It is a raw HTML content
  2. html5lib: Specifying the HTML parser we want to use.
Some important functions which useful in web scrapping:
  1. find( ): Finding out the first tag with the specified name or id and returning an object of type bs4. E.g. find(name, attrs, recursive, string, limit, **kwargs)
  2. findAll( ): Finding all tags with the specified tag names or id and returning them as a list of type bs4.  E.g.  find_all(name, attrs, recursive, string, limit, **kwargs)
  3. find_parents( ): These search methods use to iterate over all the parents and check each one against the provided filter to see if it matches. E.g. find_parents(name, attrs, recursive, string, limit, **kwargs)
How to scrape data from multiple pages?



How to scrape links and text from the website?


Other tools for Web scraping:

  1. Scrapy: Scrapy is a fast high-level web scraping framework, used to crawl websites and extract structured data from their pages.                                                                                                     To install scrapy: pip install scrapy  or conda install -c conda-forge scrapy
  2. Selenium: The selenium package is used to automate web browser interaction from python.     To install selenium: pip install selenium.
Conclusion:  In this article, we learned how to scrap data from a website using BeautifulSoup library.

         Thank you very much for reading.😊  Please read other articles.





Comments

Post a Comment

If you have any doubt, please let me know. To check my other blog kindly check the following links:
https://pythoholic.blogspot.com/

If you are interested in reading Marathi stories and other stuff, kindly check the following link.

https://pratilipi.page.link/q8dZ4ffZwKPHUx6R9

ꜰᴏʀ ᴇxᴘʟᴏʀɪɴɢ ᴛʜᴇ ᴡᴏʀʟᴅ ᴘʟᴇᴀꜱᴇ ʜᴀᴠᴇ ʟᴏᴏᴋ ᴀɴᴅ ꜰᴏʟʟᴏᴡ.
https://maps.app.goo.gl/jnKyzdDpKMFutUqR7

Popular posts from this blog

How to convert PDF file into audio file?

Image
To convert PDF files into an audio file, we use PyPDF2 and Pyttsx3 libraries. First, we use PyPDF2 for reading text from PDF and then use Pyttsx3 to convert extracted text into audio files. Let’s start with the PyPDF2 library. PyPDF2: PyPDF2 is a library in python that is used to read text from PDF. A Pure-Python library built as a PDF toolkit. It is capable of extracting document information, splitting documents page by page, merging documents page by page, etc. Pyttxs3: Pyttx3 is a text-to-speech library. It has many functions which will help the machine to communicate with us. It will help the machine to speak to us. How to install the above libraries? Pip install PyPDF2 Pip install pyttsx3 Example: import pyttsx3,PyPDF2 pdfObj = open('sample.pdf','rb') pdfreader = PyPDF2.PdfFileReader(pdfObj) speaker = pyttsx3.init() for page_num in range(pdfreader.numPages):          text = pdfreader.getPage(page_num).extractText()  ## extracting text from the PDF     clea
Read more

Pillow Libary in Python.

Image
  Pillow library:    It is a lightweight image processing tool that aid in editing, creating, and saving images. Pillow supports many image file formats including BMP, PNG, JPEG, and TIFF.  Python Imaging Library is a free and open-source additional library for the Python programming language that adds support for opening, manipulating, and saving many different images file formats. How to install pillow: p ip install Pillow The most important class in the Python Imaging Library is the Image class. Defined in the module with the same name. You can create instances of this class in several ways.   How to load an image from a file? Open(): You can use an open () function in the Image module.  object. You can now use instance attributes to examine the file contents.  To read files from disk, use the open () from function. You don’t have to know the file format to open a file. The library automatically determines the format based on the contents of the file. from PIL import Image img
Read more

How to perform operations on emails and folders using imap_tools?

Image
  IMAP_TOOLS: imap_tools is a library used to perform operations on emails and folders. Installation: pip install imap-tools How to log in to the mail?  We need to pass the username and password to the login function. And also we can pass the initial_folder name. How to get subjects from a specific folder?   First searches email nums by criteria in the current folder, then fetch and yields. E-mail attributes:  You can fetch the required data of e-mail using email attributes. Like you can fetch sender of mail using from_ attribute. You can easily fetch the date of mail using date attributes.  There are lots of e-mail attributes, we see some of them. Search criteria: You can use three approaches to build search criteria.  Search using subject. Search using text. Search using charset. AND, OR, NOT, Header and UidRange these classes are used for search logic. How to perform actions on email? You can perform the following operations on email:  copy move delete flag append How to perform act
Read more