Posting forms with Python

When you are automating your website tests, you will probably run into forms that need testing sooner or later. Even better (or worse), you wont be able to automate your tests at all if a simple login post is needed to access your site and you are unable to do so.

In python there are several approaches to post web-forms. In fact, there are too many to choose from. Therefore we created a list of the most used approaches, mention their differences and speak our preference. We dont go into much depth, but just show a simple example (and, if able, also prints the forms found on the page) and discuss the pro’s and cons.

The discussed libraries/approaches are:

  • basic urllib2 post support
  • Mechanize
  • Twill
  • ClientForm

There is also a library called Zope , able to post forms. The approach is much like the one from Mechanize, so I will not discuss that one here. When you are using pycurl, you probably end up using a different approach much like the urllib2 post support. An example for posting forms with pycurl can be found here.

Urllib2 post support:

import urllib, urllib2
page = ''
raw_params = {'usern':'test_name','passw':'test_pass'}
params = urllib.urlencode(raw_params)
request = urllib2.Request(page, params)
page = urllib2.urlopen(request)
info =

If this works for you, and you dont find yourself using alot of different forms, this simple approach is the way to go, and there is no need to start using another library.

Pros: Simple, no other library needed

Cons: Very limited flexibility and support. Unable to detect any forms on a requested page. No success or failure feedback of a post.


import mechanize
br = mechanize.Browser()
	forms = br.forms()
	print 'Forms:'
	for form in forms:
		print 'form', form
		# Force try using the expected form
		br["usern"] = "test_name"
		br["passw"] = "test_pass"
		response = br.submit()
		content = response.get_data()
	except urllib2.HTTPError, e:
		sys.exit("%d: %s" % (e.code, e.msg))
	except IOError, e:
		print e

Pros: Easy to use,  Can detect forms on page.

Cons: Limited flexibility, Can choke on bad html.


from twill.commands import go, showforms, formclear, fv, show, submit
print "Forms:"
	# Force try using the first form found on a page.
	fv("1", "usern", "test_name")
	fv("1", "passw", "test_pass")
	content = show()
	print 'debug twill post content:', content
except urllib2.HTTPError, e:
	sys.exit("%d: %s" % (e.code, e.msg))
except IOError, e:
	print e

I have found that this library is the most robust against incorrect formatted html. It supports several ‘html cleanup libraries’ and is tidying up html by default. In some cases it was able to detect forms where others could not. The flexibility is still limited though.

Pros: Easy to use. Can detect forms on page. Robust against bad formatted html.

Cons: Limited flexibility, Not the fastest.


import ClientForm, urllib2
request = urllib2.Request(page)
response = urllib2.urlopen(request)
forms = ClientForm.ParseResponse(response, backwards_compat=False)
print 'Forms:'
for form in forms:
	print form
	# Force try using the expected form
	form = forms[0]
	form["usern"] = "test_name"
	form["passw"] = "test_pass"
	content = urllib2.urlopen(
except urllib2.HTTPError, e:
	sys.exit("%d: %s" % (e.code, e.msg))
except IOError, e:
	print e

As you can see, ClientForm can detect forms on a page and specifically select a form. Trying to set a value will only work if it that operation is really possible, so it tests itself already basically.

Pros: Great flexibility and support. Most comprehensive documentation and examples.

Cons: Html cleanup advised before form detection.

Last words (Conclusion, if you will):

I have found that most of the libraries are not very robust against incorrect html format. At time of writing I have found for example that a simple incorrect hr tag (<hr/> instead of <hr /> or <hr>) can break the detection of the form. a study with the incorrect hr tag showed that only twill was able to successfully detect the form, and none were able to do a successful post. It is therefore recommended to prettify your markup before ‘feeding’ it to your library. Although not very fast, the library you want to have a look at to do this is: BeautifulSoup.

Concluding I can say that, if urllib2 post support is sufficient, there is no need using other libraries. However, once the forms you start testing get more divers and the html pages contain incorrect html markup, you will find yourself struggling with posting forms where ClientForm gives a clear overview of what forms are available and knows all the possible fields and options.
So, due to the clear examples and documentation for ClientForm, and the biggest support and freedom for form field settings, this is definately my preferred library for posting forms.

Finally, make sure you support cookies, or e.g. trying to log in wont have the desired effect. The way to go here is using cookielib. An old but still useful example on usage can be found here.

Author: Mark Barzilay

Graduated with honors from TU Delft in 2007 studying Electrical Engineering and Media & Knowledge Engineering. Founded spriteCloud in 2009 and worked on test automation ever since, helping out small and large companies with their test automation strategy and infrastructure. Mark is also leading the development on, an online platform for all your automated test results.

We’re spriteCloud, a leader in software and cybersecurity testing.

Aside from interesting articles, we also have a team of software testers that can help your organisation.

Have a look at our testing solutions.

Stay up-to-date on all things quality assurance, test automation, and cybersecurity.

Subscribe to our mailing list!