Automated Smoke Testing: PhantomJS vs. Selenium

Smoke testing is a superficial level of testing conducted by developers to make sure that all the functions of an application work. They do not go into the finer details of the app, focusing rather on the major functionalities. The developers perform smoke testing on the build before and after they give it to the testing team.

The term “smoke test” was initially coined after the hardware tests that engineers did to check if their system worked, i.e. there was no fire or smoke when the initial switch was activated. In smoke testing of an app, all areas of the application will be tested, but without going into the more nitty-gritty details.

Smoke Testing - to Check if there is Fire and Smoke

Smoke testing is also known as build acceptance testing or BAT. It is a critical aspect of Quality Assurance and delivers decisive answers on whether to accept the build or reject it. Therefore, a build in software indicates that a particular model is still in its testing stage.

All the critical functionalities would be put to test here. Once the build clears the first state, it will be taken for further QA tests, and if the build fails the test, it would be rejected and the QA team would revert it back to a previously accepted version or reject it completely.

Normally, a smoke test is performed before any critical functional tests are executed on the software built. A simple example of a smoke test would look like this:

You design a simple project where you have modules like login, view user, user detail page, task creation and new user creation. Each one of these five modules is tested to check whether it is working well, and to verify that there is a smooth transition from one module to the other.

There is another term, known as sanity testing, that is often misunderstood for smoke testing. Sanity testing is done on a software build after it has gone through the smoke testing stage and changes to the build have been made. This testing is done to ensure that the bugs are fixed and there are no persisting problems.

What is the Importance of a Smoke Test?

Testing your web applications to ensure quality is important. Conducting the smoke test is even more important as it answers basic questions such as, “Does the program run?”, “Is the logging process smooth?”, “What happens when you click the main button?” etc.A major benefit of running a smoke test is that it provides quick feedback within a matter of few minutes and testers don’t have to wait hours to get results.

Integration Testing

Integration testing is a process in which the different software modules in the computer are merged and tested as a group. For example, if your task is to test the keyboard of the computer, the process is called unit testing, but when you have to test the keyboard and the mouse, it is known as integration testing.

In the same way, when all the components of the app are joined together, integration testing is performed by the developer, the tester or by both of them.

There are different types of integration testing:

• Top-down testing - where the testing starts right at the beginning and follows down the hierarchy.
• Bottom-up testing hierarchy - where the testing starts right at the bottom of the program and works its way up.
• Big-bang testing approach - where all the components are linked together and simultaneously tested; it’s done when all the modules are ready.
• Sandwiched testing approach – a hybrid testing approach combining top-down and bottom-up approaches.

There is a basic difference between smoke testing and integration testing. Smoke testing is all about concentrating on the main components of the program. If there is a problem with any component, then the testing would be halted.

In integration testing, the test is done with different components to see how they work with one another. It is a chain; so, only when one component works normally, the other component will also be able to start working.

Take the same keyboard example, for instance - if your keyboard is working fine, but there is a problem with the monitor, you will not be able to see the end result. It is important to ensure that all the components work in accordance with each other.

What is a Headless Browser? How is it Different from a Real Browser?

Performance monitoring is conducted to detect problems with the performance and functionality of apps so they don’t impact end users. One way to do this is through synthetic testing.

In synthetic testing, you can simulate real users by loading all the critical pages like logging in, performing simple duties and checking out. This kind of synthetic testing mimics real usage, so you can gauge the typical behavior of the users.

Synthetic testing comes in two different formats: Headless Browsers, a variation of browser simulation, and Real Browsers.

Browser simulation takes place when the user requests an HTML page, parsing out and requesting resources like images, JavaScript, CSS and so on. However, it does not actually involve JS engine, so all the additional requests that normally accompany it will be ignored.

The headless browser is a variation of browser emulation. They are browsers meant for automated testing, and come with the regular GUI or Graphical User Interface. Not having a GUI calls for certain benefits like quick execution of scripted automation, it is less resource intensive and lightweight.

Real browsers are the browsers everyone is familiar with. Chrome, Safari, Firefox, Opera, Internet Explorer and the like are all known as real browsers. They are equipped with GUI and several other features that would lead to a wonderful browsing experience.

It is these features that make the real browsers more resource intensive but heavy and thus slow when executing automated commands. But they have something in common with headless browsers. You can change the user-agent string when you are working with real browsers too, making it possible to imitate other browsers when competing with servers.

The Advantage Headless Browsers have over Real Browsers

The main advantage that headless browsers have over real browsers is that the former is much faster. Real browsers take time to open because there is rendering of HTML, CSS, images and JS, while headless browsers have no such issue. Headless browsers start working immediately as there is no time wastage over page loading.

Automated smoke tests are conducted with the help of headless browsers. There are several headless browsers for JavaScript testing frameworks, out of which we will be discussing PhantomJS and Selenium in detail.

PhantomJS

This is an incredibly powerful tool for testing as it supports several web standards. This makes it a suitable option for automation and integration with CI tools. PhantomJS gives you the advantage of capturing screenshots so you can overrule the user-agent string header to the concerned server and imitate the next browser.

Being a headless version of WebKit, it works as a windowless command line browser.

Here’s a simple JavaScript example:

console.log('Loading a web page');
var page = require('webpage').create();
var url = 'http://phantomjs.org/';
page.open(url, function (status) {
//Page is loaded!
phantom.exit();
});

Credits: http://phantomjs.org/

That PhantomJS works as a single version of WebKit maybe a disadvantage of the framework. But the good news is that it performs as a self-contained command line application.

The reason why developers are very keen on using PhantomJS is that it doesn’t need many resources and absolutely no third party devices or external services. A downside to that is that you cannot use PhantomJS for extensive web app testing.

PhantomJS is perfect for smoke testing web apps. The reason why developers like to use PhantomJS is that it blends smoothly into the development workflow. PhantomJS is suitable for networking monitoring, thereby analyzing the inspection of network traffic.

You can hook PhantomJS during a request-response cycle to collect data about a particular website. It is possible to format the data to analyze the performance of a webpage.

PhantomJS is an important tool for unit testing.p PhantomJS is also used for automatic web performance test. It can run a lot of tests and these would be shown on the command line.

Advantages of PhantomJS at a Glance

• Development is very easy with PhantomJS, you can write the code by selecting the HTML element while developing the UI.
• CasperJS, a navigation scripting and testing utility for PhantomJS is used, making it easier for developers and testers.
• With PhantomJS, tests are done in 1/3rd of the time.
• Testers who need to do manual testing find it easier with PhantomJS.
• Works on all major operating systems like Windows, Mac, OS X, Linux and so on.
• It’s completely headless on Linux, also runs on Iron.io, Heroku, and Amazon EC2.
• Jenkins and YSlow makes it possible to automate performance analysis.
• Monitor page loading and export as all standard HAR files.

It is natural to think that a web browser without a graphical interface is useless, but as PhantomJS provides Javascript API, it becomes useful. This is how the phrase, “WebKit scriptable” came in. To install PhantomJS, visit the official website and get the binary suitable for your requirement.

Selenium

Selenium is an open source automation tool and a JavaScript run automated testing framework that enables you to run tests across different browsers. The most commonly used browsers are Chrome, Firefox, Safari and Internet Explorer.

Developers choose Selenium when they need a specific JavaScript engine. As Selenium is integrated with the support of a number of leading web browsers, it is considered as the main technology for automation tools, frameworks and a huge number of APIs.Also, you need a corresponding driver when you are using a specific browser with Selenium. During the test run, Selenium will launch the corresponding browser known as “in script” and goes through each of the stages in the test.

You can download Selenium Server which is distributed as a single jar file, that looks like this:

java -jar selenium-server-standalone-2.28.0.jar

Once you execute this command, a server where the testing code will be connected will immediately be booted up. You need to run Selenium Server to run the tests.

There are two versions of Selenium, known as Selenium IDE and Selenium WebDriver. Check them out:

Selenium IDE (Integrated Development Environment)

Integrated with complete Selenium Core, it allows developers to easily and quickly record and playback tests in the actual environment that they will be working in.

However, it is implemented as Firefox extension, enabling you to record, edit and debug tests. Selenium IDE is not a mere recording tool; it is the entire IDE, making it easier for you to have the perfect environment for running tests.

Selenium IDE should ideally be used only as a prototyping tool and would not be suitable for creating and maintaining complex test suites. In order to make complete use of Selenium IDE, you need to have a little bit of programming knowledge and must be familiar with JavaScript, DOM and HTML.

Selenium WebDriver

With the inclusion of WebDriver API, you have a new version of Selenium known as Selenium WebDriver. It makes direct calls to the browser by using the browser's native support for automation. Hence, it is a good choice for browser-based automation tests.

Selenium Server, featured with built-in grip capabilities is integrated within Selenium WebDriver. The best way to choose Selenium tool would be for scaling and distributing scripts across different browser environments.

WebDriver being a web-based testing tool was meant to cover all the limitations of Selenium IDE. Direct calls are made to the web browser and the test script would be executed in this style. Browser supports and capabilities are used with WebDrivers.

Selenium ensures that tests are closer to reality because it allows for automated control of real browsers on real operating systems. The framework is capable of running on various browsers and operating systems.

But when you are working with a single machine, you cannot run more than a few browsers. Selenium bindings control PhantomJS, making it possible for testing 90% of the requirements.

PhantomJS or Selenium – Which One to Select?

Selenium does have a downside when compared to PhantomJS. It needs a user-friendly graphical desktop for running the tests, making the process slow. However, you can leverage the full potential of Selenium when you set up virtual machines for various operating systems and connect them all together.

What developers love about Selenium is its capability to use bindings that would allow them to use common programming languages like Java (the native language of Selenium), JavaScript, Python, Ruby and C#.

PhantomJS running on WebKit engine is a much smaller project, but without the graphical portion and hence, faster and easier to set up. The advantage PhantomJS has is that if the application runs on WebKit, then it will definitely run on all the browsers.

With PhantomJS bindings to Selenium, it is possible for Selenium to control PhantomJS eliminating the need for any complicated set up. It allows you to cover 90% of the testing requirements and when you need to do a more powerful testing, you can add extra browser connections to Selenium without changing any of the codes.

The differences between PhantomJS and Selenium are probably obvious to you by now. But it is ultimately the developer or tester who decides which one to go for

The Final Points

We can safely conclude that PhantomJS and Selenium are two of the best JavaScript testing tools presently used by developers. As both are perfect in their roles, understanding their unique capabilities would help you decide which one to choose.

If your intention is to run multiple tests within a short period of time, then using PhantomJS would be a feasible option because with Selenium, it could be a time-consuming process.

Another advantage with PhantomJS is that as it is backed by command-line based usage, the JS code of the website is sandboxed and not left on the screen. Therefore, it doesn’t interact with the principal tool.

On the other hand, PhantomJS wouldn’t be a good choice for web app testers as it serves as a single version of WebKit.

Thus, if you prefer to use headless browsers for testing, you can choose either PhantomJS or Selenium, depending on your requirements. There is no superior framework as such; it depends on the user.

You can ensure the quality of your app with the help of an independent testing team. Our expert test engineers are ready to help you!