API Testing: How to Do It Using Runscope

Not so long ago, our QA team faced a new challenge. We had to acquire knowledge in the field of API testing so we could cover new projects with API tests. Now we are going to share our experience with you

Compatibility with various platforms, which is accomplished with the REST API on the backend side, is one key to the success of modern apps. So it’s only natural that we wanted to develop our expertise in this area. The proper performance of an API is critical for the overall quality of products, whether they be for the Web, iOS, or Android.

Of course, before we consider API testing further, we need to define what API testing is and what benefits it gives.

At Yalantis, we perform Graphical User Interface (GUI) testing on all of our projects. But while GUI testing is primarily focused on detecting bugs that show up in real-world use cases, API testing is concerned with detecting bugs that affect the business logic of the application.

What are the benefits of API testing?

#1 We begin API tests much earlier than GUI automation and manual tests. Thus, we can find severe issues at early stages of project development.

#2 API testing is language-independent because all data is converted to XML or JSON for transfer over the internet. Because of this language independence, testers can work with any language that supports XML or JSON technologies: Java, JavaScript, Python, Ruby, PHP, etc.

#3 During API testing you create testing data that’s useful for further GUI testing.

#4 API tests, in addition to functional testing, help you to improve performance and security.

As we can see, there are plenty of reasons to perform API testing on your projects.

How to perform API testing?

To perform API testing, we first need to create API documentation that’s  complete and clear. You can place all the required documentation in Apiary. This is a useful tool for creating APIs and prototyping. Apiary supports automatic document creation and provides tools for API debugging and automated API testing. We can create a list of test cases using test design techniques based on that documentation.

API testing requires adding new types of checks in addition to our traditional ones. So we performed Syntax testing, Testing method and operations, and Test scenarios.

Syntax testing

We perform syntax testing to check if a method or operation accepts correct input and rejects incorrect input.

We might test to check if:

  • Leaving mandatory fields empty results in an error.

  • Optional fields can be both empty or filled without causing an error.

  • Filling fields with incorrect data types (for example, putting a text value into an integer field) results in an error.

Testing methods and operations

We test methods and operations to check whether they function as they should, and write test cases based on the Equivalence Partitioning technique. At this step, we have to avoid any dependencies. This type of testing can be described as Unit testing, meaning that each test case should be directed to one particular function or method.

Let’s see how this works by looking at the logic of a real app.

Yalantis developed a hiring application that connects employers with qualified employees and offers such methods as login and logout, job posting, job creation, and the ability to retrieve lists of jobs that a user has already created.

We should check if each method works correctly:

  • Are data stored correctly when an Employer saves a new job using a setter method?

  • Does calling a getter method retrieve the correct information when we try to retrieve a list of previously posted jobs?



Test scenarios

After each method and operation are tested, method calls can be strung together to emulate business processes. After we’ve checked that all of our functions work correctly, we can create a test scenario that will include all of our previous checks and implement them step-by-step. You can see examples of such scenarios in the image above.

What is Runscope?

Runscope is a web-based service for creating and supporting API tests. It also provides services for monitoring API performance. With Runscope, you can write simple test scenarios for various checks of API functionality even if you’ve never written program code before.

This service can be integrated with a lot of other services including CircleCi, GitHub, Heroku, Jenkins, and TeamCity with the use of Webhook.

You can find more information about how to integrate Runscope in the following resources:

What does the process of API testing and monitoring with Runscope look like?

Here’s how our project structure looks like in Runscope.

Projects in Runscope are called ‘Buckets’. A Bucket represents a Test Run and may contain several Test Scenarios, collectively termed ‘Test Suits.’ Each of these Test Suits is composed of test steps.

Each test step is an API request that contains a set of assertions that we need to check.

To begin testing, we first need to create a new Bucket and set it up.

Trigger URLs are used to integrate with other services. You can manually add parameters to be used for calculations.

You can also import existing set of tests in the following formats:

Or you can create new Test Suits on your own.

After we’ve created a Bucket, we can set up our test environment. During the testing process, we can use one URL numerous times. So we can add it just once to our environment variables to avoid manually entering it over and over.

For example, for our hiring application, we can add to our environment variables a variable for the path to the API, a variable for test emails (both for employees and employers), and a variable referencing a list of new jobs.


Each of our tests can be run using different environment settings, which can be extremely useful if we need to check the functionality of different API versions or servers.

At this point our bucket has been created and set up, we’ve designed our tests, and we’ve saved our test cases in TestRail. Now it’s time to transfer them to Runscope.

Let’s imagine that we have to write a test case that checks if a user, who has never created a job, actually has no saved jobs. To do this, we need to perform the following actions:

1. Send a request to an API. According to our documentation, this request has to contain the required token variable in the header. Earlier we added it to our environment variables (the token variable was generated in a previous response by the server)

2. Add assertions:

  1. Check if the response has the correct status

  2. Check that we don’t have any job id’s in the body of our response

3. If both these criteria are met, the test case was executed successfully

4. For other cases, we add a script that saves all found jobs in a variable

More words about scripting in Runscope

You can use predefined JS functions and objects as well as supported by Runscope’s JS interpreter ones.

You can use functions from libraries included to interpreter:

For example, in our case function _.pluck() can be really handy if we need to get the list of workers applied for our job and save their ids in the variable.

Don’t forget to validate your scripts before adding them to tests. For example, you can use JSLint validation tool for that.

For every test case, we can set up a schedule with an execution period from one minute to one day.

With the use of Runscope, API testing is quick, and we get notifications immediately about each test failure via one of our Slack channels.

In summary, here’s a list of Runscope`s pros and cons.


  1. Useful FAQ. All features are documented fully, so you can find an answer to almost all your questions.

  2. You don’t need to know any programming language to create assertions  – just pick from the list of prepared checks.

  3. If you’re  not satisfied with the default assertions, you can write your own scripts using JS to perform your own.

  4. Versioning. You can view a timeline of your failures and successes to discover when something went wrong.

  5. Runscope uses several servers to send responses. And each of them is situated in a different place. This can be really handy when you need to check the response time of distant servers.


  1. You can’t add attachments to your queries.

  2. We found several minor issues during tool exploitation: for example, in some cases, test cases that were imported from Postman weren’t executed according to the predefined schedule.

  3. Runscope costs a lot and offers only a short trial period.

After weighing all the pros and cons each team can decide if Runscope is suitable for them or they need to search for alternatives.

Source link

Leave a Reply