Test Frameworks

Engineers write test cases to ensure that software works as expected. Often test cases require common services. These services are also agnostic of each test. The purpose of test frameworks is to provide a common platform for writing, organizing and executing test cases.

A test framework is a set of guidelines or rules that enable more efficient testing. To say it differently, a test framework provides a consistent interface between your code and your tests. Test frameworks basically provide the scaffolding. Test engineers can therefore focus on writing tests and testing the core functionality of their software.

The term Test Automation Framework is commonly used, though test frameworks can in rare cases be used to aid manual testing.

Discussion

  • What are some benefits of using a test framework?

    Some benefits include improved test efficiency, lower maintenance costs, minimal manual intervention, maximum test coverage, code reuse, handling of recovery scenarios, and easy reporting.

    Obviously, test frameworks play an essential role in test automation, and test automation increases the depth and scope of testing. Lengthy or complex tests are hard to run manually. Developers can also focus on creating more tests than spend time executing tests. Essentially, better testing leads to higher quality software in a short time to market.

    It's recommended that you select a framework already available in the market rather than write your own custom framework. By doing this, you can focus on writing effective test cases and test suites.
  • What's the typical execution process of a test framework?
    Katalon Studio uses listeners to complement setup/teardown calls. Source: Katalan Studio Docs 2018.
    Katalon Studio uses listeners to complement setup/teardown calls. Source: Katalan Studio Docs 2018.

    Here we describe the typical process. Further details will vary depending on features that frameworks support.

    The process starts by gathering all tests for execution. These tests are commonly grouped within test suites that are manually specified or could be discovered by convention. For example, all method/function names starting with a certain prefix are treated executable tests.

    Test executions start and end with setup and teardown code respectively. This setup/teardown code could be at test suite level, module level, test case level, or all of these.

    Within each test, one or more assertions are invoked. These compare current runtime status with expected status. If as expected, test execution continues. If not as expected, test fails. Stack trace may be recorded for later analysis. Extra logs may be taken. In all cases, final verdict from test execution is recorded.

    The final stage is reporting. Reports in preferred formats will contain summary, test verdicts, test-level execution metrics, and access to logs or traces.
  • What features should I look for when choosing a test framework?
    Test framework block diagram. Source: Flenner 2017.
    Test framework block diagram. Source: Flenner 2017.

    As in any tool, you should look at industry adoption, cost, performance, stability, scalability, (community) support, documentation, and features. Here are some features expected of a good framework:

    • Agnostic: Agnostic of System Under Test (SUT), automation libraries, platform or OS.
    • Language: What scripting languages are supported?
    • Usage: Easy to install, configure and use.
    • Integration: Integrates well with automation tools, Continuous Integration (CI) tools, issue tracking tools, etc.
    • Reporting: Generates detailed reports and analytics on test executions. Captures failure logs. Collects execution metrics.
    • Customization: Extend the framework. Group, tag, order or skip tests. Configure test environment.
    • Modular: Clear separation of framework layer and automation test suite. Automation infrastructure should follow coding conventions and design patterns. Promotes reuse of modules or libraries.
    • Data: Test scripts are separated from the data. Support for parameterized testing.
    • Discovery: Discover tests in project paths and automatically compose test suites for execution.
    • Playback: Record steps of manual tests and play them back for automated testing.
  • What are some types of test frameworks?
    Common types of test frameworks. Source: Devopedia.
    Common types of test frameworks. Source: Devopedia.

    Test frameworks differ in their approach to testing, which are briefly described as follows:

    • Linear: Tests are independent of one another, often created by recording manual actions and replaying them.
    • Module Based: The application is tested in its smaller parts (functions, modules, sub-systems). Tests can be combined to enable system-level tests as well.
    • Library Based: Reusable libraries are identified and written. Good for test maintenance and scalability.
    • Data Driven: Focus is on executing a test with different input data. Data is read from databases or spreadsheets. There's a clear separation of test scripts and input data. Hard-coding of data within scripts is avoided.
    • Keyword Driven: An extension of data-driven approach. Test execution is basically a sequence of steps defined by keywords. Keywords are mapped to objects and actions in a shared object repository. Promotes reuse.
    • Behaviour Driven: Tests are written in a descriptive style so that even business stakeholders can understand them. Tests are in terms of behaviours and user-focused scenarios.
    • Hybrid: A mix of the above.
  • Could you name some test frameworks out there?
    A comparison of some test frameworks. Source: Anderson 2017.
    A comparison of some test frameworks. Source: Anderson 2017.

    A blog post on DZone (March 2018) identified ten open source test frameworks: Selenium, Carina, EarlGrey, Cucumber, Watir, Appium, RobotFramework, Apache JMeter, Gauge and Robotium. Selenium is mainly for web apps while Appium is for mobile apps. Carina can use the automation of Selenium or Appium. Watir is based on Ruby while RobotFramework is based on Python. Watir supports data-driven testing while RobotFramework uses keyword-driven approach for acceptance testing.

    Cucumber is a framework for Behaviour-Driven Development (BDD). Tests are written in Gherkin language and the focus is on acceptance testing.

    Another list on TestBeacon includes Serenity, Cypress, RobotFramework, RedwoodHQ, Sahi, Galen, Gauge, Citrus, and Karate-DSL. Serenity uses Selenium WebDriver and BDD tools such as Cucumber and JBehave. Cypress enabled Test-Driven Development (TDD). Galen is useful for testing layouts and user experience. Gauge might suit TDD and BDD practitioners. Citrus is suitable for testing network calls such as REST or SOAP.

    Specific to Android testing are Appium, Robotium, Selendroid, MonkeyTalk, Calabash, UI Automator, Magneto, Kobiton, Squish, SeeTest, KMAX, MonkeyRunner, Frank, KIF, and Testdroid.

    Wikipedia has a long list of test frameworks.
  • How is a test automation tool different from a test framework?

    There's a difference between a test framework and a test automation tool. An automation tool provides the implementation of testing actions. For example, Selenium WebDriver implements actions that mimic user actions on user interfaces. Thus, Selenium WebDriver is an automation tool.

    However, Selenium WebDriver can be called from within any framework that supports it. For example, pytest is a test framework in Python, and since Selenium has Python support, we can call Selenium WebDriver APIs from Python scripts that are managed by pytest. Tools such as Selenium IDE is a simple framework that has the automation part built into it.
  • How is a test management tool different from a test framework?

    A test management tool is more about managing the testing process. This includes interfacing to various tools and processes that manage requirements, plan product releases, do reporting, track defects, execute tests, etc. There are plenty of such tools in the market. A few of them include qTest, PractiTest, Zephyr, Test Collab, and TestFLO for JIRA.

    A test framework is more about writing, managing and executing the tests. They are also about test automation. While frameworks may integrate with other tools and processes, their focus is more on testing rather than the bigger picture.

Milestones

1989

At WyattSoftware, Gary Goldberg manages his test cases in spreadsheets. This is for a financial software written in Smalltalk. His colleague Ward Cunningham writes a framework to read the tests and execute them. This is probably one of the earliest test frameworks.

1994
A SUnit test suite with three tests and results. Source: Beck 1994.
A SUnit test suite with three tests and results. Source: Beck 1994.

Kent Beck writes SUnit for unit testing of Smalltalk code. It has only three classes (TestCase, TestSuite, TestResult) and twelve methods.

1997

Inspired by SUnit, Kent Beck and Erich Gamma create JUnit for Java unit testing.

2004

At ThoughtWorks, Jason Huggins creates what's now called Selenium 1 Core to automate web app testing. It uses scripts encoded in HTML tables. Selenium is today managed by the Software Freedom Conservancy.

2012

At the Selenium Conference 2012, Dan Cuellar gives a lightning talk about automating iOS app testing using Selenium-style syntax written in C#. Later that year, he open sources the code and adds supports for Python. Jason Huggins adds Selenium WebDriver support. At the Mobile Testing Summit in November, it's launched as Appium. In 2016, Appium is handed over to JS Foundation.

References

  1. Aebersold, Kirsten. 2018. "Test Automation Frameworks." SmartBear Software. Accessed 2018-07-06.
  2. Anderson, Brian. 2017. "Best Automation Testing Tools for 2018 (Top 10 reviews)." Medium, October 26. Accessed 2018-07-06.
  3. Appium. 2018. "The History of Appium." Accessed 2018-07-11.
  4. Beck, Kent. 1994. "Simple Smalltalk Testing: With Patterns." Accessed 2018-07-11.
  5. C2 Wiki. 2003. "Framework History." June 21. Accessed 2018-07-11.
  6. Colantonio, Joe. 2018. "9 top open-source testing automation frameworks: How to choose." TechBeacon, March 06. Accessed 2018-07-06.
  7. Cotter, Sean and Michael J. Potel. 1995. "Testing." From Chapter 14 of Inside Taligent Technology, Addison-Wesley. Accessed 2018-07-11.
  8. Dasari, Sheshajee. 2014. "How To Design An Effective Test Automation Framework." Blog, Evoke Technologies, October 21. Accessed 2018-07-06.
  9. Flenner, Yoni. 2017. "20 Fundamentals for a Successful Test Automation Infrastructure." Blog, Test Project, March 26. Accessed 2018-07-06.
  10. Fowler, Martin. 2006. "Xunit." January 17. Accessed 2018-07-11.
  11. Guru99. 2018. "Top 10 Mobile Testing Tools In 2018 for Android & iOS." Accessed 2018-07-10.
  12. Hammant, Paul. 2014. "Happy 10th Birthday, Selenium." Blog, ThoughtWorks, October 22. Accessed 2018-07-11.
  13. Helppi, Ville-Veikko. 2015. "The Basics Of Test Automation For Apps, Games And The Mobile Web." Smashing Magazine, January 14. Accessed 2018-07-06.
  14. Ihnatsyeu, Dzmitry. 2018. "Improve Your Selenium WebDriver Tests with PyTest." Blog, BlazeMeter, April 13. Accessed 2018-07-10.
  15. Katalan Studio Docs. 2018. "Test Listeners (Test Hooks)." Katalon Studio, June 08. Accessed 2018-07-11.
  16. Kazeeva, Anastasiya. 2018. "10 Best Open Source Test Automation Frameworks for Every Purpose." DZone, March 14. Accessed 2018-07-06.
  17. Mindfire Solutions. 2018. "A Look at Some Open Source Android App Testing Tools." Medium, June 19. Accessed 2018-07-10.
  18. Pytest Docs. 2018. "Changing standard (Python) test discovery." v3.6.3, May 23. Accessed 2018-07-11.
  19. Reminnyi, Sasha. 2015. "Developing Test Automation Scripts and Automation Frameworks." InfoQ, October 24. Accessed 2018-07-06.
  20. Sharma, Lakshay. 2014. "Behavior Driven Development." ToolsQA, December 26. Accessed 2018-07-10.
  21. Shebanow, Andrew. 2007. "A Brief History of Test Frameworks." shebanator blog, August 21. Accessed 2018-07-11.
  22. Software Testing Help. 2018a. "Most Popular Test Automation Frameworks with Pros and Cons of Each – Selenium Tutorial #20." Software Testing Help, June 07. Accessed 2018-07-06.
  23. Software Testing Help. 2018b. "Top 15 Best Test Management Tools in 2018 (Our Reviews)." Software Testing Help, July 03. Accessed 2018-07-06.
  24. Testbytes. 2017. "5 Major Types of Test Automation Frameworks." Testbytes, July 17. Accessed 2018-07-06.
  25. TestingWhiz. 2017. "6 Features to Look for in an Ideal Test Management Tool." TestingWhiz, July 24. Accessed 2018-07-06.
  26. Twist Team. 2010. "Test Assertions - How do they work?" Blog, ThoughtWorks, October 19. Accessed 2018-07-11.

Further Reading

  1. Aebersold, Kirsten. 2018. "Test Automation Frameworks." SmartBear Software. Accessed 2018-07-06.
  2. Reminnyi, Sasha. 2015. "Developing Test Automation Scripts and Automation Frameworks." InfoQ, October 24. Accessed 2018-07-06.
  3. Helppi, Ville-Veikko. 2015. "The Basics Of Test Automation For Apps, Games And The Mobile Web." Smashing Magazine, January 14. Accessed 2018-07-06.
  4. Beck, Kent. 1994. "Simple Smalltalk Testing: With Patterns." Accessed 2018-07-11.

Article Stats

Author-wise Stats for Article Edits

Author
No. of Edits
No. of Chats
DevCoins
User avatar lokesh.rawat
3
0
2273
Avatar of user arvindpdmn arvindpdmn
4
0
71
DevCoins due to articles, chats, their likes and article hits are included.
1379
Words
2
Authors
7
Edits
0
Chats
5
Likes
19K
Hits

Cite As

Devopedia. 2019. "Test Frameworks." Version 7, January 5. Accessed 2024-06-25. https://devopedia.org/test-frameworks
Contributed by
2 authors

Last updated on
2019-01-05 03:47:35

Improve this article

Article Warnings

  • In References, replace these sub-standard sources: guru99.com