Automation Testing Frameworks: How To Choose?

Automation Testing Framework is a set of tools and libraries for test automation. Frameworks can be supported by one or more programming languages such as JavaScript, Java, Python or C#.

Test Framework orchestrates and runs tests, sets a unified test format, reports results, conveniently defines test data (constants, datasets, etc.), and combines libraries specific to testing a particular product.

Test Framework has many advantages at its disposal. If well-designed, it increases code reuse, simplifies test maintenance, makes tests more readable, unifies reporting, and makes running tests on different environments easier.

When choosing an automated test framework, consider the following factors:

• project requirements;

• budget;

• technology stack;

• level of team skills;

• functionality and capabilities of the framework.

It is important to remember that there is no ready-made solution. However, knowledge and experience will be the best advisers. Discover in-depth insights into various types of automation testing and the importance of browser compatibility testing in ensuring seamless user experiences. Dive into our articles to explore comprehensive guides, best practices, and case studies (like the InspectHOA) that shed light on these critical aspects of software testing.

The methods for creating and running automated tests divide into coded and non-coded. Code-based automation system uses programming languages to write test code, giving more flexibility and possibilities for complex and customizable tests. Non-code automation uses a graphical interface and blocks to create tests without using user code, which is simpler and faster but can be limited in functionality and flexibility for complex tests. The chosen approach depends on the team's needs and the project. Some teams want maximum flexibility and control, others - the quick creation of simple tests.

Test automation environments are based on several criteria: structure, design principles, functions, or degree of abstraction. So, test automation frameworks can be roughly divided into five types:

• Linear Scripting Framework;

• Test Library Architecture Framework;

• Data-Driven Testing Framework;

• Keyword-Driven or Table-Driven Testing Framework;

• Hybrid Automation Testing Framework.

We propose to consider the types of automation frameworks and their functionality in more detail.

A linear scripting framework is the simplest and most basic test automation framework thatuses Selenium WebDriver to record and play test actions in your browser. Here, test cases are described as a linear sequence of actions and consist of several steps the tester must perform in a specific order.

After each step, the expected result is checked, and the success or failure of this test case is revealed. A linear scripting environment is easy to create and run but challenging to maintain and reuse. The linear scripting environment suits small projects with simple, stable requirements and minimal test scenarios. Plus, it can be helpful in exploratory testing or proof of concept.

But there is the other side of the coin. Its use in large-scale projects with complex and dynamic requirements and multiple test scenarios can lead to code duplication, high maintenance costs, poor readability, and scalability.

TLAF is a software that uses test script libraries for automation. It is a tool to make it easy to create large test libraries for the Robot Framework using Python. This framework allows you to create and reuse unit test script libraries across different test projects. They operate on the principle of calling functions and returning test results. Not to mention that they are flexible and scalable, adaptable to different situations, and easy to integrate with other automation tools such as version control and automated test runners.

The test library architecture platform suits medium-sized projects with moderate requirements and multiple test scenarios. However, it is unsuitable for large-scale projects with complex and dynamic requirements and different datasets.

Data-Driven (Data-Based) Testing Framework is a software testing method for creating test cases with different data sets. It generates test data and expected results, and WebdriverIO executes these scripts across different browsers and devices. This framework separates test data and scripts, allowing you to run the same test script with different data sets. Instead of embedding input data in a test script, data is provided from external sources such as spreadsheets, databases, or CSV files, making it easy to modify and extend test cases without changing the scripts. The Data-Based Testing Framework offers many benefits, such as test reusability, scalability, ease of maintenance, and broad test coverage.

Keyword-Driven or Table-Driven Testing Framework is a test automation technique that uses keywords or tables to describe test script logic. The framework is designed to develop test scenarios with different sets of data and functionality in which FitNesse is used to implement and execute these scenarios in the form of wiki pages. Keywords or tables are high-level commands. They define the checks or actions to be performed in the test and can be stored within the test script or external data sources. Such testing makes creating easy-to-read test cases independent of the code possible.

A hybrid test automation framework is a framework that combines the features and benefits of two or more frameworks. For instance, keyword-driven, data-driven, linear, or modular frameworks. Often used to develop flexible and scalable test scripts, along with Selenium WebDriver to implement and run these scripts across platforms and browsers, as well as TestNG or JUnit to organize, parameterize and report on tests.

The hybrid framework allows you to overcome the hurdles and challenges of using a single framework. In addition, it gives you more freedom and productivity when creating and running tests. Hybrid driven framework can be tailored to the specific needs and preferences of the test team and project.

To get started, remember that there are approximately 20 automation testing frameworks and tools. Among the most popular are Selenium, WebDriverIO, Cypress, Nightwatch, Playwright, and Robot Framework.

Often, client applications exist in a web or mobile version for Android and iOS.  However, they may also have a desktop version. The customer wanted to automate the tests to speed up the release, reduce bugs, increase the coverage, catch new and existing bugs qualitatively, and integrate the tests into the CI/CD. We, in turn, needed to expand the number of browsers and devices that could be tested simultaneously. This is how our team went about it:

First, we need to understand which platforms this application is designed for. Then can use one framework for all platforms or make multiple repositories. This solution depends on what different cases occur on the platforms. If the difference is only in locators, you can use one repository and, respectively, one structure. And if the thread is different, and you have to do different cases, then the frameworks may be different, the most controversial for each platform.

After we made a list of functionality and corresponding requirements for automation that the framework should provide. It may include support for a particular programming language, integration with development tools, or the ability to run tests in parallel.

Our talented automation engineers knew most languages, tools, and frameworks and could write test scripts in any language. Consider some of them from a practical perspective:

Java and C # — are now mainly used to support old projects. For new projects, JavaScript is often chosen.

Selenium — supported but outdated (wide community, proven option).

WebdriverIO — mobile, iFrames.

Cypress — all tests of one test set are performed in one browser tab, convenient and very informative reports.

Playwright frequently releases, but a new one and the community is just forming.

JavaScript/TypeScript: New projects in JS/TS (because JS/TS is popular for the front-end, there are many packs, a wide community, and the server part in Node.js).

Python: Python is gaining momentum (PyPI is also a large collection of libraries).

Selenium (web application management) + PyTest (base for writing autotests).

Playwright + PyTest.

PyTest only — API or database.

Robot framework (new gaining momentum).

Nose — a little.

Here you have to make an adjustment that in automation, you need people who are:

• Well-versed in what is called Data Science and database automation.

• They know how to choose the proper tests for automation.

• They know how to determine the right level of automation and do not try to automate everything.

• They know how to work with APIs and automate APIs and databases.

• And good at writing interface automation scripts (this is mandatory).

Having decided on languages and frameworks, let's move on.

Then, we looked at what requirements from the list above relate to automation. We created an automation strategy, which greatly helps assess the project's scope and goals and clarify the results.

For example:

1. Estimate the project scope and select functions suitable for automation.
2. Writing lists of automation scripts that you definitely cannot do without.
3. Determining the implementation timeline.
4. Understand key implementation and support tools.

So, as of now, we've figured out the scale.

At this stage, we thought about the platform on which the project would be and wrote a list of tools and frameworks. It's important to remember to define the platform first. It determines what tools will be used. Then we also picked up "tools" for APIs, databases, web, and desktop automation.

It's time to evaluate the budget of the project. It all depends on how much the tools and infrastructure cost and how much it is to support them. Next, you must consider paying for priority frameworks or working with free, open-source ones. To be honest, we used both options, depending on the situation and financial capabilities of the client. 

Next, you must consider paying for priority frameworks or working with free, open-source ones. To be honest, we used both options, depending on the situation and financial capabilities of the client. 

When it comes to open-source tools, they rely on the community for support. However, getting timely and comprehensive help isn't always guaranteed. On the upside, these tools evolve rapidly, making it worth considering investing in their development.

On the other hand, commercial tools offer many attractive features, but the prices are sometimes unpleasantly shocking. When relying on community support, the experience might not always be smooth, and additional costs could arise for tutorials or specialist training.

We kept cards for each project, and therefore, we knew what tools were used earlier by automation testers (and related specialists) in our company. We also knew what our developers had used in unit testing. Tools and frameworks differ depending on the scope of testing. So there may be different tools for testing APIs, interfaces, databases, etc.

Before starting a new project, we consider what we have worked with previously. We also look at and analyze how effective it was, how relevant it is now, and whether it is outdated. 

Finally, we made a list of questions:

• Is there a problem?

• Did everything go well?

• Could it be better?

Answers helped us make decisions.

When the framework issue was resolved, it was time to write PoC (proof of concept). This is the final clarification in the project, which clarifies the issues:

• How will the framework work in the project?

• Is it easy to work?

• Does it fully meet our requirements for the project?

• Is it suitable for priority test cases, is it flexible enough for different scenarios and application versions?

The PoC defines the risks associated with the framework and tools. Some frameworks (and auxiliary things) may have to be abandoned based on this PoC list.

Criteria for evaluating frameworks:

This is a small outline of the main criteria.

Initial data: Canadian company, Long-Term Healthcare project.

Goal: To Build automation from scratch.

What was used: One framework and one repository (same test cases), TypeScript programming language.

Considering all the inputs, we proposed the only suitable framework option - WebdriverIO.

As a result of implementing automation on the selected framework, the client received the following:

1. Increased speed. Automated testing allowed us to run a large number of tests in parallel. For example, having 2000 tests and running them in 10 threads, the execution would take 1 hour, while testing this functionality manually would require at least a week of specialist work.
2. Cost reduction. Taking point 1 into account, we could confidently say that automation could replace hundreds of hours of manual work of specialists and their equipment and save resources for other needs.
3. Accuracy. Automatic testing had high accuracy and reliability because the result did not depend on the human factor. Errors during testing were kept to a minimum.
4. Various conditions. Automated tests could be run under different conditions, on different browsers, screen sizes, OS, and connection speeds simultaneously.
5. Security. Automated tests also helped identify critical security vulnerabilities in software testing framework for web applications by covering the largest environment with tests.

This is one small example in favor of choosing the proper framework.

There are many types of automation testing frameworks, each with its own advantages and disadvantages. There is no single best framework for all situations, so careful analysis and comparison of different options is essential before making a decision. Of course, we gave you a small guide, but we still recommend contacting professionals. Contact us today to learn more about our automation testing services and how we can support your testing needs. Our team of experienced professionals is here to assist you every step of the way.