Software Performance Testing

Performance testing helps identify bottlenecks in a system, establish a baseline for future testing, support a performance tuning effort, and determine compliance with performance goals and requirements.

Performance testing is an ongoing task throughout the life cycle of the project. It not only involves testers, but also developers and operations to run and maintain the application to its performance expectations.

Performance, load, and stress tests are subcategories of performance testing, each intended for a different purpose.

Discussion

  • What is performance testing in software?

    Performance testing is a type of non-functional testing. It's intended to determine the responsiveness, throughput, reliability, and/or scalability of a system under a given workload.

    Performance testing is testing that is performed to determine how fast some aspect of a system performs under a particular workload. It can serve different purposes, such as to demonstrate that the system meets performance criteria.

    In the past, performance testing was a specialized activity managed by separate teams, not the developers. The process typically took weeks and even months. With the transition to Agile and DevOps, this has changed. Using open source and commercial tools, it's now become possible to do performance testing by developers in shorter time scales.

  • What should be the focus of performance testing?

    Following are some focus areas for performance testing:

    • Speed: Determines whether the software application responds quickly.
    • Scalability: Determines maximum user load the software application can handle.
    • Stability: Determines if the application is stable under varying loads.
  • What are the common goals of performance testing?

    Performance testing has the following goals:

    • Assess production readiness
    • Evaluate against performance criteria
    • Compare performance characteristics of multiple systems or system configurations
    • Find the source of performance problems
    • Support system tuning
    • Find throughput levels
  • What are the types of performance testing?
    Types of software performance testing. Source: Devopedia.
    Types of software performance testing. Source: Devopedia.

    Each type of performance testing has a specific focus as noted below:

    • Load Testing: Checks the application's ability to perform under anticipated user loads. The objective is to identify performance bottlenecks before the software application goes live.
    • Stress Testing: Also called Breakpoint Testing, this involves testing an application under extreme workloads to see how it handles high traffic or data processing. The objective is to identify breaking point of an application.
    • Endurance Testing: Also called Soak Testing, this is done to make sure the software can handle the expected load over a long period of time.
    • Spike Testing: Tests the software's reaction to sudden large spikes in the load generated by users.
    • Volume Testing: Large amount of data is populated in database and the overall software system's behavior is monitored. The objective is to check software application's performance under varying database volumes.
    • Scalability Testing: Determine the software application's effectiveness in "scaling up" to support an increase in user load. It helps plan capacity addition to your software system.
  • What are the steps of performance testing?
    Steps of Performance Testing. Source: Odom 2014, Steps of Performance Testing.
    Steps of Performance Testing. Source: Odom 2014, Steps of Performance Testing.

    We may list the following steps:

    • Identify the Test Environment: Identify the physical test environment and the production environment, which includes the hardware, software, and network configurations.
    • Identify Performance Acceptance Criteria: Identify the response time, throughput, and resource utilization goals and constraints. In general, response time is a user concern, throughput is a business concern, and resource utilization is a system concern.
    • Plan and Design Tests: Identify key scenarios, determine variability among representative users such as unique login credentials and search terms.
    • Configure the Test Environment: Prepare the test environment, tools, and resources necessary to execute each strategy as features and components become available for test.
    • Implement the Test Design: Develop the performance tests in accordance with the test design best practice.
    • Execute the Test: Run and monitor your tests. Validate the tests, test data, and results collection.
    • Analyze Results, Report, and Retest: Consolidate and share results data. Analyze the data both individually and as a cross-functional team. Re-prioritize the remaining tests and re-execute them as needed.
  • What performance parameters are monitored during performance testing?

    The following are commonly monitored:

    • Processor Usage: Amount of time processor spends executing non-idle threads.
    • Memory Use: Amount of physical memory available to processes on a computer.
    • Disk Time: Amount of time disk is busy executing a read or write request.
    • Bandwidth: Shows the bits per second used by a network interface.
    • Private Bytes: Number of bytes a process has allocated that can't be shared amongst other processes.
    • Memory Pages per Second: Number of pages written to or read from the disk in order to resolve hard page faults.
    • Page Faults per Second: Overall rate in which fault pages are processed by the processor.
    • Response Time: Time from when a user enters a request until the first character of the response is received.
    • Throughput: Rate a computer or network receives requests per second.
    • Maximum Active Sessions: Maximum number of sessions that can be active at once.
    • Hits per Second: Number of hits on a web server during each second of a load test.
    • Database Locks: Locking of tables and databases.
    • Thread Counts: Number of threads that are running and currently active.
  • Could you give pointers on performance testing in the cloud?
    SmartBear's LoadComplete enables on-prem plus cloud-based load generation. Source: SmartBear 2018.
    SmartBear's LoadComplete enables on-prem plus cloud-based load generation. Source: SmartBear 2018.

    Cloud-based performance testing refers to the use of cloud technologies to test your application or service. Your application itself may not be on the cloud but the load generation, report collection and test monitoring is done in the cloud. The obvious benefit is lowering infrastructure cost, testing at scale, simulating load coming from different geographic regions, and more.

    This should not be confused with cloud performance testing, where the idea is to validate the performance of an application deployed in the cloud. The general approach to performance testing, types of tests and parameters monitored remain the same but with some aspects particular to cloud. For example, storage, processing, bandwidth and number of users become important. In particular, parameters related to management of virtual machines, containers, microservices, database connections, all need to be validated.

    Tests should validate how well and fast can the cloud infrastructure adapt to varying load. Elasticity, scalability, availability, fault tolerance and reliability are all important measures. This includes the performance of logging, raising alarms, or sending notifications, across all the cloud services used by the application.
  • Could you name a few performance testing tools?
    Screenshot of CA BlazeMeter showing a graphical report of load testing. Source: Karow 2017.
    Screenshot of CA BlazeMeter showing a graphical report of load testing. Source: Karow 2017.

    Some open source tools include Apache JMeter, K6, LoadUI, and Grinder. Commercial tools requiring a license include HP LoadRunner, IBM Rational Performance Tester, and Microfocus SilkPerformer.

    When selecting a particular tool, you must evaluate the supported features. One blogger has shared a feature comparison of some tools with respect to scripting, execution and reporting.

    Wikipedia gives a useful list of load testing tools.

Milestones

1989

Mercury Interactive is founded in California. It releases LoadRunner, one of the first performance testing tools.

Dec
1998

JMeter is released as an open source functional plus performance testing tool. In November 2011, it becomes a top-level Apache project.

2001

OpenSTA is released as an open source web load and performance testing tool.

2004

Software Test & Performance magazine is launched by BZ Media. It's later renamed to Software Test & Quality Assurance magazine. A curated list of magazines and blogs for software testing is also available.

2011

Based on JMeter, commercial tool BlazeMeter is released. Fourth generation of BlazeMeter is released in 2018 and it enables Continuous Testing. BlazeMeter enhances JMeter with scalability, reporting, collaboration, and test management features.

Sample Code

  • import http from "k6/http";
import { sleep } from "k6";
 
export let options = {
  vus: 10,
  duration: "30s"
};
 
export default function() {
  http.get("http://test.loadimpact.com");
  sleep(1);
};
 

References

  1. AgileLoad. 2013. "Application performance testing." AgileLoad, February. Accessed 2018-08-27.
  2. Apache JMeter. 2018. "History and Future." JMeter User Manual. Accessed 2018-08-27.
  3. Bedell, Crystal. 2011. "Performance testing in the cloud." TechTarget, November. Accessed 2018-08-27.
  4. Girmonsky, Alon. 2018. "BlazeMeter V4 (Fourth Generation) is Coming." Blog, BlazeMeter, March 04. Accessed 2018-08-27.
  5. Guru99. 2018. " Performance Testing Tutorial: Types, Process & Important Metrics." Accessed 2018-08-27.
  6. IBM. 2020. "Start performance testing early." IBM Cloud Garage Method, IBM Corporation. Accessed 2020-07-23.
  7. Karow, Dave. 2017. "The History and Development of Performance Testing." DZone, February 08. Accessed 2018-08-27.
  8. Meerts, Joris and Dorothy Graham. 2018. "The History of Software Testing." Testing References. Accessed 2018-08-27.
  9. Microsoft Docs. 2010. "Chapter 1 – Fundamentals of Web Application Performance Testing." Performance Testing Guidance for Web Applications, Microsoft Docs, April 27. Accessed 2018-08-27.
  10. Murty, Srinivas. 2016. "Performance Testing in a Nutshell." Blog, ThoughtWorks, February 23. Accessed 2018-06-29.
  11. Odom, Phillip. 2014. "Steps of performance testing." Blog, Loadstorm, April 29. Accessed 2014-04-29.
  12. Paidi, Kranthi. 2018. "What should be your Strategy for the Performance Testing in Cloud?" Blog, Cigniti Technologies, March 11. Accessed 2018-08-27.
  13. SQA For Dummies. 2020. "Performance Testing (NFR)." SQA For Dummies. Accessed 2020-07-23.
  14. SmartBear. 2018. "Load Testing Tool for Websites and Web Apps." LoadComplete. Accessed 2018-08-27.
  15. Wikipedia. 2018a. "Software performance testing." Wikipedia, July 27. Accessed 2018-08-27.
  16. Wikipedia. 2018b. "Load testing." Wikipedia, July 27. Accessed 2018-08-27.

Further Reading

  1. Moorthy, Ramya Ramalinga. 2016. "Myths about Performance Testing vs. Performance Engineering Unveiled." DZone, September 01. Accessed 2018-08-27.
  2. TestOrigen. 2017. "Actuality versus Myths of Performance testing at various stages." TestOrigen, September 19. Accessed 2018-08-27.
  3. Cohen, Noga. 2018. "Why Load Testing is Important." Blog, BlazeMeter, June 19. Accessed 2018-08-27.
  4. AgileLoad. 2013. "Application performance testing." AgileLoad, February. Accessed 2018-08-27.
  5. K6 Docs
  6. JMeter, Get Started

Article Stats

Author-wise Stats for Article Edits

Author
No. of Edits
No. of Chats
DevCoins
User avatar narendra
9
2
907
Avatar of user arvindpdmn arvindpdmn
5
3
811
DevCoins due to articles, chats, their likes and article hits are included.
1245
Words
2
Authors
14
Edits
6
Chats
8
Likes
10K
Hits

Cite As

Devopedia. 2020. "Software Performance Testing." Version 14, July 23. Accessed 2024-06-25. https://devopedia.org/software-performance-testing
Contributed by
2 authors

Last updated on
2020-07-23 10:17:43

Improve this article
tools testing performance

See Also

Article Warnings

  • In References, replace these sub-standard sources: guru99.com
  • Readability score of this article is below 50 (46). Use shorter sentences. Use simpler words.