Exercise: URL Scraping with Python to Find Hidden Directories on a Metasploitable Machine

Step 1: Install the Required Libraries

● Install the Python libraries ‘requests’ and ‘beautifulsoup4’ by running the following

pip install requests
pip install beautifulsoup4

Step 2: Create a Python Script

● Create a new Python script and import the ‘requests’ and ‘beautifulsoup4’ libraries.
● Use the ‘requests’ library to send a GET request to the target URL and store the
response in a variable.
● Use the ‘beautifulsoup4’ library to parse the HTML content of the response and find
all the ‘a’ tags.

Step 3: Extract the URLs

● Extract the URLs from the ‘a’ tags and store them in a list.
● Iterate over the list of URLs and check if they contain any hidden directories.

Step 4: Analyze the Results

● Print the URLs that contain hidden directories and evaluate the results.

Step 5: Case Study Scenario

● Target Metasploitable machine IP:
● Target URL:
● Sample code for URL scraping:

import requests
from bs4 import BeautifulSoup

url = “”
response = requests.get(url)
soup = BeautifulSoup(response.content, ‘html.parser’)

for link in soup.find_all(‘a’):

● This exercise is only for educational and research purposes, and should not be used
to perform unauthorized activities.
● Do not use this script on a website that you do

● Python Requests library documentation: https://requests.readthedocs.io/en/master/
● BeautifulSoup4 library documentation:
● A tutorial on web scraping with Python and BeautifulSoup:
● Metasploitable Project, a vulnerable machine for testing Metasploit:
● A guide on how to use Metasploitable for penetration testing:

Open chat
Hello 👋.
Tell me, how can I help you?