In the grand theater of the World Wide Web, the dialogue between a client and a server is governed by a strict protocol. The client, typically a web browser but for our purposes a Python program, initiates a conversation by sending an HTTP request to a specific Uniform Resource Locator, or URL. The server, upon receiving this request, processes it and replies with an HTTP response. This response is a package containing several pieces of information, but two are of paramount importance: a status code and, usually, a body of content.
The status code is a three-digit number that succinctly communicates the outcome of the request. We are most familiar with codes in the 200s, particularly 200 OK, which signifies success. The content we requested, perhaps an HTML document or a JSON payload, is found in the response body. But not all requests result in the immediate delivery of the desired resource. Sometimes, the resource has moved.
This is where the 3xx series of status codes comes into play. These are the redirection codes. When a server returns a status code like 301 Moved Permanently or 302 Found, it is informing the client that the resource is no longer at the requested URL. The server is not being unhelpful, however. Accompanying this 3xx status code is a crucial piece of information in the response headers: the Location header. This header contains the new URL where the client should go to find the resource.
Amazon Basics 128 GB Ultra Fast USB 3.1 Flash Drive, Black
$14.99 (as of July 26, 2025 11:20 GMT +03:00 - More infoProduct prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on [relevant Amazon Site(s), as applicable] at the time of purchase will apply to the purchase of this product.)A well-behaved client, upon receiving a 3xx response, is expected to read the Location header and automatically issue a new request to this new URL. This second request may itself result in another redirect, leading to a chain of requests and responses, a journey from one URL to the next until, hopefully, a 200 OK response is received and the final destination is reached. To the user of a web browser, this entire sequence is often invisible, perceived as a single, seamless navigation. But for the programmer, understanding this underlying multi-step process is essential.
Let us observe this directly. The fundamental tool for opening URLs in Python’s standard library is urllib.request.urlopen. In its most basic form, it attempts to hide the complexities of redirection from us. Consider a request to a URL that is known to redirect, such as the HTTP version of a site that has migrated to HTTPS.
import urllib.request
url = 'http://github.com'
response = urllib.request.urlopen(url)
# The response object we get is from the *final* destination
print(f'Final URL: {response.url}')
print(f'Status Code: {response.status}')
When you execute this code, you will likely not see a 3xx status code. Instead, you will see a status of 200 and the final URL, which will be https://github.com/. The urllib.request.urlopen function, by default, employs a series of handlers, one of which is responsible for diligently following these redirect instructions. The initial response from http://github.com was, in fact, a redirect, but this handler intercepted it, made the new request to the URL specified in the Location header, and presented us only with the final, successful result. This automatic behavior is convenient, but it obscures the journey. To truly manage redirects, we must first acknowledge that they are happening and then seize the mechanisms that control them. The default behavior is to follow the trail of breadcrumbs left by the server; our goal is to become the master of that trail, deciding when to follow, when to stop, and when to examine the path itself. This journey from a simple request to a potentially complex chain of transactions is the reality of the modern web.
An Opener for Following the Trail
The simplicity of urllib.request.urlopen is deceptive. It is a high-level convenience function, a facade that stands in for a more complex and configurable system of objects. To gain control over how requests are processed, we must peer behind this facade and interact with the machinery directly. The core of this system is the “opener,” an object formally known as an OpenerDirector. An opener orchestrates the process of fetching a URL by dispatching tasks to a collection of “handler” objects.
Each handler is a specialist, designed to manage a specific aspect of the HTTP protocol or a related URL scheme. There are handlers for basic HTTP and HTTPS requests, handlers for FTP, handlers for proxy servers, handlers for authentication, and, most importantly for our current discussion, handlers for HTTP redirects. When you call the global urlopen function, you are implicitly using a pre-configured opener that has been installed with a default set of these handlers, including one for redirects.
The key to managing redirects, therefore, is not to avoid urlopen but to construct our own opener, populating it with the handlers we require. The module urllib.request provides a function named build_opener for this very purpose. We can start by creating an opener that does nothing but follow redirects, thus replicating the default behavior in a more explicit fashion. The handler responsible for this task is an instance of the HTTPRedirectHandler class.
Let us reconstruct our previous experiment, but this time, we will assemble the components ourselves.
import urllib.request
# First, we instantiate the handler responsible for following redirects.
redirect_handler = urllib.request.HTTPRedirectHandler()
# Next, we build an opener, passing our handler to it.
# The opener will now use this handler to process requests.
opener = urllib.request.build_opener(redirect_handler)
# We use the open method of our custom opener object.
url = 'http://github.com'
response = opener.open(url)
# The result is identical to using the default urlopen.
print(f'Final URL: {response.url}')
print(f'Status Code: {response.status}')
The output of this program is identical to that of our first example. We still arrive at the final HTTPS URL and see a status code of 200. Nothing has changed in the outcome. However, something profound has changed in our approach. We are no longer relying on a hidden, implicit mechanism. We have explicitly declared our intent to handle redirects by creating an HTTPRedirectHandler and incorporating it into an opener. This act of building our own opener is the first and most crucial step toward gaining fine-grained control over the redirection process. By choosing which handlers to install, we dictate the capabilities of our web client. We have now exposed the component that follows the trail; the next step is to modify its behavior.
Gaining Control by Subclassing the Handler
While constructing our own opener grants us the power to choose our tools, the HTTPRedirectHandler itself, in its default state, remains a black box. It performs its duty of following redirects silently and automatically. We have brought the handler into the light, but its internal operations are still concealed. To truly manage the redirection process, we need to modify the behavior of the handler itself. The Pythonic way to customize the functionality of a class is not to rewrite it from scratch, but to inherit from it and override its methods.
The urllib.request framework is designed for precisely this kind of extension. When an OpenerDirector processes a response, it inspects the status code. If the code indicates an error or a special condition—and all 3xx redirect codes are treated as a form of “error” that a handler can choose to resolve—it looks for a handler object with a method whose name matches a specific pattern: protocol_error_code. For an HTTP response with a status code of 302, the opener will search for a handler with a method named http_error_302. For a 301, it will look for http_error_301, and so on. The HTTPRedirectHandler class contains implementations for all the common redirect codes (301, 302, 303, 307, 308).
This mechanism provides the hook we need. By creating a new class that inherits from HTTPRedirectHandler and providing our own implementation of a method like http_error_302, we can intercept the redirection process at the exact moment it is about to occur. We can inspect the details of the redirect, log them, and then decide whether to allow the redirection to proceed.
Let us create a custom handler that announces each redirect it encounters before letting the process continue. We will subclass HTTPRedirectHandler and override the methods for both permanent (301) and temporary (302) redirects. Inside our overridden methods, we will print the details of the redirection before delegating the actual work back to the original implementation in the parent class using Python’s super() function.
import urllib.request
class VerboseRedirectHandler(urllib.request.HTTPRedirectHandler):
"""A redirect handler that prints the details of each redirect."""
def http_error_301(self, req, fp, code, msg, headers):
print(f"--- Redirect Encountered ---")
print(f"Status: {code} {msg}")
print(f"Original URL: {req.full_url}")
print(f"Redirecting to: {headers['Location']}")
# Delegate to the parent class to perform the actual redirect
return super().http_error_301(req, fp, code, msg, headers)
def http_error_302(self, req, fp, code, msg, headers):
print(f"--- Redirect Encountered ---")
print(f"Status: {code} {msg}")
print(f"Original URL: {req.full_url}")
print(f"Redirecting to: {headers['Location']}")
# Delegate to the parent class as well
return super().http_error_302(req, fp, code, msg, headers)
# We use a URL that is known to redirect, like our http-to-https example.
url = 'http://www.python.org'
# Build an opener with our custom, verbose handler.
opener = urllib.request.build_opener(VerboseRedirectHandler())
print(f"Requesting URL: {url}n")
response = opener.open(url)
print("n--- Request Complete ---")
print(f"Final URL: {response.url}")
print(f"Final Status: {response.status}")
When this program is executed, our custom code is invoked. The output clearly shows the message from our overridden http_error_301 method. We see the original URL we requested and the new URL provided in the Location header. Our method receives five arguments: the original Request object req, the file-like object fp containing the body of the redirect response (which is usually empty), the integer status code, the status message msg, and the headers object from the response. After printing this information, our method calls the parent class’s version of http_error_301. This call is essential; it is the original implementation that reads the Location header, constructs a new request, and returns it to the opener to continue the chain. By calling super(), we inject our logging behavior without disrupting the fundamental redirection logic. We have successfully placed a probe into the heart of the machine, allowing us to observe its inner workings. The next logical step is to move from passive observation to active intervention. What happens if we do not call the parent method? What if, instead, we return something else, or nothing at all?
A Study in Permanent and Temporary Moves
The distinction between permanent and temporary redirects extends beyond a simple suggestion to a client about caching. The specific status code used—301 Moved Permanently versus 302 Found, for example—carries significant implications for how a client should behave, particularly when the original request involves more than a simple retrieval of data. The most common scenario involves the POST method, typically used to submit form data to a server. If a client sends data via POST to a URL that redirects, should the client re-submit, or “re-post,” that same data to the new URL? The answer is complex and has been a source of ambiguity in the history of the HTTP specification.
The original 302 Found status code was not explicit about method preservation. In practice, most web browsers and other clients adopted a behavior of changing the request method from POST to GET upon receiving a 302 redirect. This meant that the data from the original POST request was dropped, and a new, safe GET request was made to the location specified in the redirect. While this behavior became a de facto standard, it was technically a deviation from the original intent. To resolve this ambiguity, later versions of the HTTP standard introduced new status codes. The 303 See Other status code explicitly instructs the client to retrieve the resource at the new location using a GET request, formalizing the common behavior. In contrast, the 307 Temporary Redirect status code was created to instruct the client that it *must not* change the request method. If the original request was a POST, the new request to the redirected location must also be a POST.
A parallel situation exists for permanent redirects. The 301 Moved Permanently status code shares the same ambiguity as the 302, and clients frequently change a POST to a GET. Its stricter counterpart is 308 Permanent Redirect, which, like 307, forbids changing the request method.
The urllib.request.HTTPRedirectHandler class correctly implements this nuanced behavior. It will convert POST requests to GET for 301 and 302 responses, but it will preserve the POST method for 307 and 308 responses. We can demonstrate this by constructing a small local server that can issue different types of redirects on command.
import http.server
import socketserver
import threading
import time
PORT = 8080
REDIRECT_TARGET = f'http://localhost:{PORT}/destination'
class RedirectingServer(http.server.SimpleHTTPRequestHandler):
def do_GET(self):
if self.path == '/destination':
self.send_response(200)
self.send_header('Content-type', 'text/plain')
self.end_headers()
self.wfile.write(b'Final destination reached via GET.')
else:
self.send_error(404, 'Not Found')
def do_POST(self):
if self.path == '/temp_302':
self.send_response(302)
self.send_header('Location', REDIRECT_TARGET)
self.end_headers()
elif self.path == '/temp_307':
self.send_response(307)
self.send_header('Location', REDIRECT_TARGET)
self.end_headers()
elif self.path == '/destination':
# Handle a POST to the final destination
content_length = int(self.headers['Content-Length'])
post_data = self.rfile.read(content_length)
self.send_response(200)
self.send_header('Content-type', 'text/plain')
self.end_headers()
response_body = b'Final destination reached via POST with data: ' + post_data
self.wfile.write(response_body)
else:
self.send_error(404, 'Not Found')
def run_server():
with socketserver.TCPServer(("", PORT), RedirectingServer) as httpd:
httpd.serve_forever()
# Run the server in a daemon thread
server_thread = threading.Thread(target=run_server, daemon=True)
server_thread.start()
time.sleep(0.5) # Allow server to start
With this server running, we can create a client that uses a custom redirect handler to observe the change, or lack thereof, in the request method. The HTTPRedirectHandler class provides a method, redirect_request, which is called internally by the various http_error_3xx methods. This method is responsible for creating the new Request object for the redirected URL. By overriding this method, we can intercept the process at a convenient, centralized point.
import urllib.request
import urllib.parse
class MethodObservingHandler(urllib.request.HTTPRedirectHandler):
def redirect_request(self, req, fp, code, msg, headers, newurl):
print("--- Redirect Intercepted ---")
print(f"Status: {code} {msg}")
print(f"Original Request Method: {req.get_method()}")
# Let the parent class create the new request object
new_req = super().redirect_request(req, fp, code, msg, headers, newurl)
print(f"New Request Method: {new_req.get_method()}")
print(f"Redirecting to: {newurl}")
return new_req
def perform_test(url_path):
url = f'http://localhost:{PORT}{url_path}'
print(f"n>>> Testing POST to {url}")
opener = urllib.request.build_opener(MethodObservingHandler())
post_data = urllib.parse.urlencode({'value': 'secret'}).encode('ascii')
try:
response = opener.open(url, data=post_data)
print("--- Final Response ---")
print(f"Status: {response.status}")
print(f"Body: {response.read().decode()}")
except urllib.error.HTTPError as e:
print(f"Error: {e.code} {e.reason}")
# Test the 302 redirect, which should change the method
perform_test('/temp_302')
# Test the 307 redirect, which should preserve the method
perform_test('/temp_307')
The execution of this program provides a clear and unambiguous result. For the request to /temp_302, our custom handler prints that the original method was POST, but the new request’s method is GET. The final response from the server confirms this, indicating that the destination was reached via GET. The POST data is lost in the process. For the request to /temp_307, however, the output shows that both the original and new request methods are POST. The handler preserves the method, and the final response from our server shows that the POST data was successfully received at the new location. This empirical result validates our understanding of the protocol and the library’s faithful implementation of its rules. The choice between redirect codes is a design decision with tangible consequences for the flow of data in a web application.
Source: https://www.pythonlore.com/managing-http-redirects-with-http-client-httpredirecthandler/
