Step 1: Install the Required Libraries
● Install the Python libraries ‘requests’ and ‘beautifulsoup4’ by running the following
commands:
pip install requests
pip install beautifulsoup4
Step 2: Create a Python Script
● Create a new Python script and import the ‘requests’ and ‘beautifulsoup4’ libraries.
● Use the ‘requests’ library to send a GET request to the target URL and store the
response in a variable.
● Use the ‘beautifulsoup4’ library to parse the HTML content of the response and find
all the ‘a’ tags.
Step 3: Extract the URLs
● Extract the URLs from the ‘a’ tags and store them in a list.
● Iterate over the list of URLs and check if they contain any hidden directories.
Step 4: Analyze the Results
● Print the URLs that contain hidden directories and evaluate the results.
Step 5: Case Study Scenario
● Target Metasploitable machine IP: 192.168.1.104
● Target URL: http://192.168.1.104/
● Sample code for URL scraping:
import requests
from bs4 import BeautifulSoup
url = “http://192.168.1.104/”
response = requests.get(url)
soup = BeautifulSoup(response.content, ‘html.parser’)
for link in soup.find_all(‘a’):
print(link.get(‘href’))
Note:
● This exercise is only for educational and research purposes, and should not be used
to perform unauthorized activities.
● Do not use this script on a website that you do
Reference:
● Python Requests library documentation: https://requests.readthedocs.io/en/master/
● BeautifulSoup4 library documentation:
https://www.crummy.com/software/BeautifulSoup/bs4/doc/
● A tutorial on web scraping with Python and BeautifulSoup:
https://www.datacamp.com/community/tutorials/web-scraping-using-python
● Metasploitable Project, a vulnerable machine for testing Metasploit:
https://www.vulnhub.com/entry/metasploitable-2,29/
● A guide on how to use Metasploitable for penetration testing:
https://www.hackingarticles.in/metasploitable-tutorial-for-beginners/
