Let's talk about testing

Posting forms with Python

When you are automating your website tests, you will probably run into forms that need testing sooner or later. Even better (or worse), you wont be able to automate your tests at all if a simple login post is needed to access your site and you are unable to do so.

In python there are several approaches to post web-forms. In fact, there are too many to choose from. Therefore we created a list of the most used approaches, mention their differences and speak our preference. We dont go into much depth, but just show a simple example (and, if able, also prints the forms found on the page) and discuss the pro’s and cons.

The discussed libraries/approaches are:

  • basic urllib2 post support
  • Mechanize
  • Twill
  • ClientForm

There is also a library called Zope , able to post forms. The approach is much like the one from Mechanize, so I will not discuss that one here. When you are using pycurl, you probably end up using a different approach much like the urllib2 post support. An example for posting forms with pycurl can be found here.

Urllib2 post support:

import urllib, urllib2
page = ''
raw_params = {'usern':'test_name','passw':'test_pass'}
params = urllib.urlencode(raw_params)
request = urllib2.Request(page, params)
page = urllib2.urlopen(request)
info =

If this works for you, and you dont find yourself using alot of different forms, this simple approach is the way to go, and there is no need to start using another library.

Pros: Simple, no other library needed

Cons: Very limited flexibility and support. Unable to detect any forms on a requested page. No success or failure feedback of a post.


import mechanize
br = mechanize.Browser()
	forms = br.forms()
	print 'Forms:'
	for form in forms:
		print 'form', form
		# Force try using the expected form
		br["usern"] = "test_name"
		br["passw"] = "test_pass"
		response = br.submit()
		content = response.get_data()
	except urllib2.HTTPError, e:
		sys.exit("%d: %s" % (e.code, e.msg))
	except IOError, e:
		print e

Pros: Easy to use,  Can detect forms on page.

Cons: Limited flexibility, Can choke on bad html.


from twill.commands import go, showforms, formclear, fv, show, submit
print "Forms:"
	# Force try using the first form found on a page.
	fv("1", "usern", "test_name")
	fv("1", "passw", "test_pass")
	content = show()
	print 'debug twill post content:', content
except urllib2.HTTPError, e:
	sys.exit("%d: %s" % (e.code, e.msg))
except IOError, e:
	print e

I have found that this library is the most robust against incorrect formatted html. It supports several ‘html cleanup libraries’ and is tidying up html by default. In some cases it was able to detect forms where others could not. The flexibility is still limited though.

Pros: Easy to use. Can detect forms on page. Robust against bad formatted html.

Cons: Limited flexibility, Not the fastest.


import ClientForm, urllib2
request = urllib2.Request(page)
response = urllib2.urlopen(request)
forms = ClientForm.ParseResponse(response, backwards_compat=False)
print 'Forms:'
for form in forms:
	print form
	# Force try using the expected form
	form = forms[0]
	form["usern"] = "test_name"
	form["passw"] = "test_pass"
	content = urllib2.urlopen(
except urllib2.HTTPError, e:
	sys.exit("%d: %s" % (e.code, e.msg))
except IOError, e:
	print e

As you can see, ClientForm can detect forms on a page and specifically select a form. Trying to set a value will only work if it that operation is really possible, so it tests itself already basically.

Pros: Great flexibility and support. Most comprehensive documentation and examples.

Cons: Html cleanup advised before form detection.

Last words (Conclusion, if you will):

I have found that most of the libraries are not very robust against incorrect html format. At time of writing I have found for example that a simple incorrect hr tag (<hr/> instead of <hr /> or <hr>) can break the detection of the form. a study with the incorrect hr tag showed that only twill was able to successfully detect the form, and none were able to do a successful post. It is therefore recommended to prettify your markup before ‘feeding’ it to your library. Although not very fast, the library you want to have a look at to do this is: BeautifulSoup.

Concluding I can say that, if urllib2 post support is sufficient, there is no need using other libraries. However, once the forms you start testing get more divers and the html pages contain incorrect html markup, you will find yourself struggling with posting forms where ClientForm gives a clear overview of what forms are available and knows all the possible fields and options.
So, due to the clear examples and documentation for ClientForm, and the biggest support and freedom for form field settings, this is definately my preferred library for posting forms.

Finally, make sure you support cookies, or e.g. trying to log in wont have the desired effect. The way to go here is using cookielib. An old but still useful example on usage can be found here.

4 responses to “Posting forms with Python”

  1. Andrew Pelt says:

    Cool Thanks for this blog. I am new at django and this is a big help.

  2. Jeff Terry says:

    Cool Thanks for the article. I am a newbie at python and this will be a big help.

  3. Mark says:

    Thanks for the comments. Any feedback regarding the post is more than welcome!

  4. on this day time and age, I undoubtedly can’t believe which i am truly creating something such as this in a really review, however it could possibly be the reality. In this game, your fingers slide over the screen to tilt, slide, jump, and turn through the blockades and courses that are as difficult as they are addicting. Google Maps is one of the best free Android apps available to all users of Android phones.

Reputation. Meet spriteCloud

Find out today why startups, SMBs, enterprises, brands, digital agencies, e-commerce, and mobile clients turn to spriteCloud to help improve their customer experiences. And their reputation. With complete range of QA services, we provide a full service that includes test planning, functional testing, test automation, performance testing, consultancy, mobile testing, and security testing. We even have a test lab — open to all our clients to use — with a full range of devices and platforms.

Discover how our process can boost your reputation.

We use cookies to give you the best browsing experience possible. For more information, please read our cookie policy.

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.