Testing Phone Fields- Part I- US Phones

Phone number fields are without a doubt the most time-consuming thing I have ever tested.  This seems so unlikely- phone numbers are standardized to ten digits in the US, so how can they cause so many headaches?  The answer is that, as we saw with Zip codes, there are so many ways to get them wrong.

If you are fortunate enough to be testing a new application without any existing data, you can fend off most phone number headaches by clearly establishing validation rules for phone number fields.  The easiest type of validation is one that accepts ten digits only, and no other characters.  The developer can rely on UI formatting to put the number in an easily-readable format rather than expecting the user to do it themselves.  The reason why this is the best course of action is that allowing the user to add in formatting creates all sorts of ambiguity.  If my phone number is 800-867-5309, how might I type that in?  Just with dashes, or will I use parentheses, such as (800) 867-5309?  What about the older format of 1-800-867-5309?  With that format, I now have eleven digits.  

Many developers try to accommodate all styles in the mistaken notion that this will make things easier for the user, but this actually makes things much more complicated.  If both parentheses and dashes are allowed, there needs to be some sort of regex used for the validation.  Otherwise, a user could type in something like ((800)--867 5309 and the number will be accepted.  If the developer you are working with has decided that he or she would like to make it easy on the user by accepting parentheses and dashes, try entering values like 800--867--5309, 800(867)5309, and 8-0-0-867-5309.  After you have demonstrated that it's possible to enter a badly formatted phone number, the developer's strategy may change.

As with Zip codes, it's important to consider how the phone number is saved to the database.  The best strategy for US phone numbers is to strip out any formatting and save the number as just the ten digits.  That way the number can be displayed in any way the UI designer deems appropriate.

Next, it's time to think about how bad phone numbers will be displayed to the user.  If a number has been saved with formatting, will it be displayed with that formatting?  Will some attempt be made to strip the formatting?  Will it be displayed at all?  I have seen all kinds of bad numbers in databases.  One of my favorites was "dad's office number".  No one can format that!  A good strategy would be to format any ten-digit number by stripping out all of the non-number characters, and then adding in whatever format is desired for display.  If the phone number is eleven digits and starts with a 1, the leading 1 could be stripped off and then the number could be formatted for display.  With any other values, the "number" could be displayed as-is, or it could not be displayed at all.  Whatever is decided, it's important to test with whatever bad numbers are currently in the database. 

Another wrinkle not often considered is phone extensions.  Many offices still use extensions to direct a call to a specific person.  Sometimes phone fields allow extensions as part of the number, but this is a bad idea, for the same reasons that allowing the user to format their own number is a bad idea.  Phone extensions can be any number of digits, so allowing them as part of the phone number means that now the validation can't expect a specific number of digits.  Also, how might the user indicate that an extension is part of the number?  Will they enter 800-867-5309 ext. 1234?  800-867-5309x1234?  Allowing these types of variations will mean allowing letters and other characters, which it makes it more difficult to validate.  A far better solution is to include a separate field for an extension.  If your developer expresses interest in including the extension as part of the phone field, some testing with entries like 8008675309extextext...1234 will probably dissuade them. 

Finally, remember to test editing phone numbers, especially numbers that are in an incorrect format.  What will happen when the user has a phone number like 800-867-530 and tries to edit another field on the form?  Will they be notified upon saving that the number is incorrect and needs fixing?  This can be a good way to involve users in cleaning their own data.  What should happen when the user tries to fix an invalid number?  Ideally they should be able to save the value with a new valid number. 

I hope that this post has emphasized the importance of having good phone field validation.  If you are fortunate enough to be testing a brand-new application with an empty database, taking good care to ensure that only valid phone numbers are saved to the database, you will prevent many testing headaches in the future! 

This post only tackled United States phone numbers.  In my next post, I will take on the even greater challenge of international phone numbers! 

Testing Date Fields

Date fields are another data type that seems simple to test.  After all, dates are standard throughout the world: there's a month, a day of the month, and a year.  But as you will see below, there are many factors to consider and many scenarios to test.

There are three main areas to think about when testing a date field:

• What format will be accepted?
• How will the date be stored in the database?
• How will the saved date be displayed?  

There are many ways to format a date when entering it into a form.  One very important factor to consider is whether the system is expecting an American format: month, day, year; or a European format: day, month, year.  For example, if I try to enter next Wednesday's date (12/13/2017) into some forms, it won't be accepted if the European format was what the form was expecting, since there is no 13th month.  And if I enter July 4 into this form as 7/4/18, it may be saved to the database as April 7th.  

Beyond the American/European formatting, there are still a myriad of ways to format dates.  For example, will all four digits of the year be expected, or is it possible to put in just the last two digits?  Will single-digit months and days be allowed, or is it necessary to precede those digits with zeroes, such as in 07/04/2018?  What about spelling out the month, as in December 13, 2017?  The developer should have a clear idea of what format or formats will be allowed, and should clearly communicate that to the user.  For example, a tooltip such as "mm/dd/yyyy" can be used inside the field to help the user know what format to use.  Regardless of whether the tooltip is in place, it is important to test with a number of different date formats to ensure that an appropriate error message is displayed when the expected format is not followed.  Also be sure that the date itself is checked for whether it is a valid date; for example, 2/30/2016 should never be allowed, nor should 18/18/18.  

Next, let's consider how the date will be stored in the database.  Some developers, not knowing better, might store the date as a string, rather than as a datetime value.  This is nearly always a bad idea for a number of reasons.  If the accepted format changes over time, or if good validation was not in place in the past, there could be strings where the date is saved as "7/4/2017" and strings where the date is saved as "December 13, 2017".  This will make it difficult to consistently display the data.  Also, if the dates are stored as strings it will be difficult to sort correctly by ascending or descending date.  "2017-12-13" might wind up coming before "3/7/2017" in an ascending sort.  As testers, we should verify that the date is stored as a datetime value, and advocate for a change if it is not. 

It's also important to look at how dates are displayed.  When a user calls up saved information, they should be able to read the dates easily. "2017-12-13T00:00:00" might be the way the date is saved in the database, but a user won't be able to interpret this date quickly.  The developer or designer should decide what date format would be best for display purposes and should stick to it consistently throughout the application.  It's also important to consider what should happen in the case of bad data.  What if there is a date saved in the database that says simply "December 13"?  Should it be displayed as 12/13/0000?  Should it not be displayed at all?  These are important scenarios to consider and test.  

There is one final consideration to make when testing date fields, and that is what the upper and lower limits of the date are.  For example, are dates in the future ever allowed?  What about dates from 100 years ago?  Remember that the future and the past change every day!  Let's say that our application doesn't allow dates in the future.  That means that today, 12/13/17 is not an allowed date in our application.  But by next Wednesday, 12/13/17 will be allowed.  And of course, you may be reading this blog post years in the future, at which time 12/13/17 will be a thing of the past!  

Testing Postal Codes

A text field with a postal code looks so simple, and yet it can be one of the most complex things to test on a form.  There are two important questions to ask before you start testing postal codes:

1. What countries does our application support?

2. What formats will we be accepting for the postal codes?

It always comes as a bit of a relief when I find out that the application I am testing will only be supporting US Zip codes.  Zip codes traditionally come in a five-digit format, such as 10012.  But there are also Zip +4 codes to consider, such as 10012-1234, and this is where the second question comes into play.  Will the application be accepting Zip +4 codes?  Will it be requiring the hyphen between the first five digits and the next four digits, or will a space be accepted as well?  What about just nine straight digits, with no hyphen or space in between?  If nine straight digits will be accepted, it's important to also verify that 6, 7, and 8 digits will not be accepted.  

And there is another very important thing to test with US Zip codes: the leading zero.  There are many fields which will strip leading zeros off of a number upon submission.  It's very important to submit some Zip codes with leading zeros and verify that they have been saved correctly to the database.  

The next likely postal code type you may encounter (if you are based in the US) is the Canadian postal code.  All Canadian postal codes are six characters in this pattern of letters and numbers:  A1A 1A1.  It's important to clarify with the developer whether the space between the two groups of three characters will be expected, or whether you can submit the code with no space.  Hopefully the validation will expect the correct letter-number pattern, and will reject postal codes like 1A1 A1A.

What if your application expects both US and Canadian postal codes?  If this is the case, the third important question to ask is:

3. Will there be a separate validation pattern for each type?

If there is a separate validation pattern, the code may first look to see what country the address contains, and then use the appropriate validation.  In this case, it's a good idea to test that you can't choose "United States" for your country and then add "A1A 1A1" as the postal code.  Or the validation pattern may be chosen by the number of characters submitted.  If six or seven (including the space) characters are submitted, a Canadian validation pattern could be used.  If five, nine, or ten (including the hyphen) characters are submitted, a United States validation pattern could be used.  Understanding what validations the developer is using will allow you to craft appropriate test cases.  For example, in a scenario where only US and Canadian postal codes are used, there should never be a scenario where eight characters are accepted.  

When we move into postal codes for other countries, things can get more confusing.  Many countries have five-digit codes, which are easy to validate.  Other countries, such as Russia, have six-digit codes.  But consider Great Britain, which has postcodes in two sections: the first section can have between two and four characters, and the second section always has three characters. There is a space between the two sections, and the postcode will always start with a letter, not a number.   In this case, it's best if the developer has ways to recognize if a British postcode is being used (perhaps by looking at what country has been entered in the form) and uses a separate validation regex for those codes.  For testing, be sure to try several different valid postcodes from various places in Great Britain, with two, three, and four characters in the first section.  You can also test with codes that have the right number of characters, but have the space in the wrong place, or with a code that has a number as the first character.

Finally, remember to check that the valid postal codes that you have submitted have been saved correctly to the database, and remember to verify that retrieved postal codes are displaying correctly.  Also be sure to check that any invalid codes in the database (that may have been added before the correct validation was in place) are displaying as added, or not at all, and that it's possible to edit both invalid and valid postal codes.  

I hope that this post has helped you see that postal codes are not as simple as they seem!  Asking the three questions listed above can help you and your developer to recognize potential issues even before testing has begun.  

Automating CRUD Testing

There are a number of different patterns we can use to automate CRUD testing.  At the very minimum, we want to test one operation of each: Create, Read, Update, and Delete.  For the purposes of this discussion, let's assume the following:

1. We are testing the simple form used in this post
2. We are doing UI automation (API automation will be discussed in a future post)

This is the pattern I like to use when testing CRUD.  I'm writing in Specflow/Cucumber for simplicity:

Scenario: Adding a user
Given I am adding a new user
When I add a first name and save
Then I navigate to the Users page
And I verify that the first name is present

Scenario: Updating a user
Given I am updating a user
When I change the first name and save
Then I navigate to the User page
And I verify that the first name has been updated

Scenario: Deleting a user
Given I am deleting a user
When I delete the first name and save
Then I navigate to the Users page
And I verify that the first name is not present

These three tests have tested Create, Update, and Delete. The first two tests are also testing Read, because we are retrieving the user for our assertions.  Therefore, with these three tests I'm testing the basic functionality of CRUD.

Some may argue that these tests are not idempotent.  Idempotency means that a test can be run again and again with the same results.  I can't run the third test over and over again with the same results, for example, because after the first time I run it, the user is no longer there to delete.

If we wanted to solve the idempotency issue, we could write a test like this:

Scenario: CRUD testing of user
Given I am testing CRUD
When I add a first name and save
And I verify that the first name is present
When I change the first name and save
And I verify that the first name has been updated
When I delete the first name and save
Then I verify that the first name has been deleted

This one scenario tests all the CRUD functionality of the form.  But it also has three different assertions.  I prefer to keep my UI tests down to one or two assertions.

When my original three scenarios are looked at as a group, they are idempotent together.  My tests are responsible for creating and deleting my data.

It would also be a good idea to test some negative scenarios with our CRUD testing, such as creating a user with an invalid first name, and updating a user with an invalid first name.  These tests could look like this:

Scenario: Creating a user with an invalid first name
Given I am adding a new user
When I enter an invalid first name and save
Then I verify that I receive the appropriate error message on the page
And I navigate to the User page
And I verify that the user has not been added

Scenario: Updating a user with an invalid first name
Given I am updating an existing user
When I update the first name with an invalid value and save
Then I verify that I receive the appropriate error message on the page
And I navigate to the User page
And I verify that the existing first name has not been updated

The first scenario is idempotent, because nothing is actually being added to the database.  The second scenario is also idempotent, but it requires an existing user.  We could assume that our user will always be present in the database, but if someone inadvertently deletes it, our test will fail.  In this case, it would be good to add in a Before step and an After step that will create the user at the beginning of the test suite and delete it when the suite is over.

These five scenarios- the three to test the happy path, and the two negative tests- would be a great regression suite for our simple form.  This is a very simple form, with just one field, which is not exactly a real-world scenario.  But it is a good way to start thinking about automated UI test patterns.

CRUD Testing Part II- Update and Delete

In last week's post, we started looking at CRUD testing.  As you recall, CRUD stands for Create, Read, Update, and Delete.  Last week we discussed testing Create and Update operations, and now we will continue by looking at Update and Delete.

In our discussion of the Read operation last week, I mentioned how important it is to test scenarios where the data in the database is invalid.  This is also true for Update operations.  Just because a text field is supposed to be required and have a certain number of characters doesn't mean that's how it is in the database!

Below is a matrix of testing scenarios for editing a text field.  Remember from last week that our hypothetical text field has the following validation rules:  it is a required field, it must have at least two characters, it must have 40 or fewer characters, and it should only have alphanumeric characters or hyphens and apostrophes; no other symbols are allowed.  As with the Create operation, be sure to test that the newly edited field is correct in the UI and in the database after the update.

For the bad values that you are testing with, be sure to vary them so you are covering a number of different validation scenarios.  For the good values that you are testing with, make sure to test hyphens and apostrophes, numbers and letters, and the upper and lower limits of the character count. 

Finally, let's discuss Delete.  The main thing to test with Delete is that the value has been deleted from both the UI and the database.  But just as with Read and Update, you'll want to make sure that you can delete bad values.  For example, if your First Name field has forty-one characters in it (violating the rule that only 40 characters or less should be added), you'll want to make sure that you can delete it in the UI.  

You may be wondering how to find all these invalid values for testing.  While it is possible to find them by searching through the existing records, the easiest way to test them is to put them in yourself!  This is why it's important to know the appropriate query language, such as SQL, for your database.  

For those who are wondering why testing with bad values is important, I once had a scenario where I was testing contact information for a person.  There were a number of incorrectly formatted phone numbers in the database.  When I tried to update any of the contact information for the person, I got the message that the record couldn't be updated, because there was invalid data present.  This was true even if I was trying to update the invalid phone number in question!  Please be sure to check for scenarios like these; your end-user will thank you!

CRUD Testing Part I- Create and Read

In spite of its unappealing name, CRUD testing is extremely important!  CRUD stands for Create, Read, Update, and Delete.  As any tester knows, much of our testing involves these operations.  Today we'll discuss the best ways to test Create and Read. 

The most important thing to know about testing CRUD is that it's not enough just to rely on what you are seeing in your UI to confirm that a field's value has been created or changed.  This is because the UI will sometimes cache a value for more efficient loading in the browser.  What you need to do to be absolutely sure that the value has changed is to check the database where your value is stored.  So you'll be confirming that your value is set in two places: the UI and the database.  If you are doing API testing as well, you can actually confirm in three places, but we'll save discussing API testing for another post.

Testing a Create Operation:

This text field looks similar to the one we looked at last week, but now we know what it does!  This is a form to add a user.  We'll enter the first name of the user into the text field and click Submit.  Next, we'll take a look at the Users page of our imaginary application and verify that the new user is present:

And there it is!  Finally, we need to query our database to make sure that the value has saved correctly there.  In our imaginary database, this can be done by running

SELECT * from Users

This will give us a result that looks a lot like what we are seeing in the UI, so I won't include a screenshot here.

To thoroughly test the Create function, you can use some of the same ideas that we talked about in last week's post.  Verify that valid entries of all kinds are saved to the database.

Testing a Read Operation:

We actually started testing the Read operation when we checked the Users page to verify that our new user was added.  But there is something else that is important to test!  We need to find out what happens when bad data is in the database and we are trying to view it in the UI.

Let's take a look at what some bad data might look like in the database:

In our imaginary application, there are some constraints in the UI for the First Name field.  It is a required field, it must have at least two characters, it must have 40 or fewer characters, and it should only have alphanumeric characters or hyphens and apostrophes; no other symbols are allowed.  As we can see in our table, we've got lots of bad data:

• User 2 has no entry for First Name
• User 3 has an empty string for a First Name
• User 4 is violating the rule that the name must have at least two characters
• User 5 is violating the rule that the name must have 40 or fewer characters
• User 6 is violating the rule that only hyphens and apostrophes are allowed for symbols

What should happen when we view the Users list in our application?  That will depend on what the product designers decide.  They may choose to display bad data as long as it is not a security risk, such as the First Name for User 6, which is actually a Stored XSS attack.  Whatever the rules for display are, it's important to test that those rules are respected.  

You may be saying to yourself (or a developer may be saying to you), "Displaying bad data won't be an issue, because we are putting good validation in place to make sure that bad data won't get in the database to begin with."  While this is absolutely standard practice today, there will always be cases where bad data will slip in.  I once tested a PATCH operation where phone numbers could be inserted into a record.  I discovered that while validation was taking place when the PATCH body was formed correctly, there was an edge case where a PATCH body was formed incorrectly and accepted without validation.  Any developer that incorrectly coded the PATCH operation could inadvertently allow bad phone numbers into the database!  

A good rule of thumb for testing any Create and Read operation is to assume that anything can go wrong and test accordingly.  We'll continue imagining what can go wrong next week, when we test Update and Delete operations.  

Testing a Text Field

A text field in an application seems so ordinary, and yet it is one of the most important things we can test.  Why?  Because text fields provide an entryway into an application and its underlying database.  Validation on a text field is what keeps lousy data from getting into the database, which can cause all sorts of problems for end users and engineers. It can also prevent cross-site scripting attacks and SQL injection attacks. 

There are a myriad of ways to test a text field, and I will be outlining several in this post.  First, let's imagine that we are testing the text field with absolutely no information about what it does:

• Click Submit without filling in the text field
• Press the space bar several times in the text field and then click Submit
• See how many characters you can fit in the text field and then click Submit (an excellent tool to count characters is https://lettercount.com)
• Fill the field with as many numbers as you can and then click Submit
• Add a negative sign, fill the field with as many numbers as you can and then click Submit
• Enter every non-alphanumeric field on the keyboard and click Submit.  If you get an error, see if you can narrow down which key (or keys) is causing the error.
• Enter in non-ASCII characters and emojis and click Submit.  If you get an error, see if you can narrow down which symbol (or symbols) is causing the error.
• Try cross-site scripting by entering in this script: <script>alert("I hacked this!")</script>  If on Submit, you get a popup message, then you know the field is vulnerable to cross-site scripting.
• Try a SQL injection attack, such as FOO'); DROP TABLE USERS; -- (Don't try this on your Production database!)

Next, let's assume that you have some knowledge about what is supposed to be entered into this text field and what the boundaries are on the data:

• Try putting in a value that is a different data type from what is expected; for example, if this text field is expecting a value of currency, try putting in a string or a date
• If the field is expecting a string, try putting in a string with one fewer characters than is expected, one more character than is expected, the lower limit of characters that is expected, the upper limit of characters that is expected, and twice the maximum number of characters expected
• If the field is expecting a numeric value, try putting the maximum value, the minimum value, a value above the maximum, a value below the minimum, and a value twice the maximum value
• If the field is expecting an integer, try submitting a value with a decimal point
• If the field is expecting a float, try submitting a value with two decimal points and a value that begins with a decimal point
• If the field is expecting a value of currency, try submitting a value with more than two digits after the decimal point
• If the field is expecting a date, try putting in the maximum date, the minimum date, one day over the maximum date, one day before the minimum date, and a date one hundred years above or below the limit
• For date fields, try entering a date that doesn't make sense, such as 6/31/17 or 13/13/17  (There are many more ways to test date fields; I'll touch on this in a later post)
• If the field is expecting a time, try entering a time that doesn't make sense, such as 25:15
• If the field is expecting a phone number, try entering a number that doesn't conform to the expected format (There are many, MANY more ways to test phone numbers; I'll touch on this in a later post as well)

For all of the above tests, find out what sort of error message you are supposed to be receiving, and verify that you are getting the correct message.

Finally, let's think about automation.  Assuming you have very thoroughly tested your text field manually, it's probably not necessary to automate every single one of your tests.  Moreover, most forms have more than one text field, and having many tests for each individual field could result in many time-consuming tests.  But here are some suggestions for what you might want to automate:

• submitting a null value
• submitting an empty string 
• submitting a value that meets the criteria (the "happy path")
• submitting the maximum number of characters or maximum value
• submitting the minimum number of characters or minimum value
• submitting just above the maximum characters or value
• submitting just below the minimum characters or value

This list of tests is not meant to be completely exhaustive; it's just a way to get you to start to think about the vast number of tests you can run on a single field.  Don't ever assume that the developer who coded the field has put in the appropriate validation; check it for yourself!  I tested a date field once where there was a limit put on the year that could be entered so it couldn't be before 1900 or after the present year.  While I did receive the appropriate message when I entered a value of 1880, I discovered that I could enter a date from the year 1300!

Think Like a Tester

Beginning with this week's post, my blog will be taking on a new focus!

I have renamed it from Fearless Automation to Think Like a Tester (for the moment, the URL will remain the same). There were three recent events that made me decide to shift my focus:

• I attended a large international computing conference where there was not a single workshop or presentation focused on software testing. 
• At this conference, I met computer science students who asked me if there were any college classes to learn to be a tester.
• I interviewed a QA engineer who was able to create a great automated testing solution for a website, but could not think of simple manual tests for the site. 

All of these things made me realize the following:

• There aren't enough people talking about testing software
• There aren't enough resources to learn about testing software
• The testing community has been focused for so long on how to test software that we haven't been thinking about what to test and why we are testing it 

Testing is truly a craft, and one that requires a different skill set from software development:

• Rather than thinking of ways to make software work, testers think of ways to make software break
• Rather than designing things to go right, testers think of all the ways that things can go wrong
• Rather than focusing deeply on one feature, testers focus on how all those features integrate
• Rather than solving a problem and moving on, testers come up with ways to continually verify that features are working 

In the weeks and months to come, I will be getting back to basics and discussing all areas of software testing- manual and automated- that require thinking like a tester. Hopefully both testing newbies and seasoned testers alike will find this knowledge helpful!

API Testing vs. UI Testing

Recently someone asked me “If you have API testing, you don’t need UI testing, right?”  I said “No, because you need to have tests that make sure that elements such as buttons are present and working correctly.”  

Then he asked, “Then if you have UI testing, you don’t need API testing?”  I said, “No, because UI tests tend to be slow and flaky.  You can get more tested in less time with API testing.”

Inspired by that conversation, I thought I’d share my thoughts on when you should do API testing and when you should do UI testing.

First, test as much as you can with API testing.  Take a look at all of your possible endpoints and  create a suite of tests for each.  Be sure to test both the happy path and the possible error paths.  On every test, assert that you are getting the correct response code.  For GET requests, assert that you receive the correct results.  If there are filtering parameters you can pass in with the request, be sure to test scenarios with and without those parameters.  For POST, PUT, and PATCH tests, test that the changes you made have been written to the database; you can do this with a GET.  Be sure to test scenarios where you are entering invalid data; assert that any message returned in the body of the response is the correct message.  For DELETE requests, test that the resource has been deleted from the database; this can be verified with a GET. 

Once you have tested all the scenarios you can think of with API testing, then it’s time to think about UI testing.  First consider your most common user story.  For example, if you are testing an address book, the most likely scenario for a user would be adding in a new address.  You could create a UI test that would navigate to the address book, click a button to add a new address, add the address, save it, and then search the address book to verify that it has been saved.  

Now that your most common user story has been added, you have probably touched a number of the elements that you would want to verify in your UI.  Next, think about other elements on the page that you might want to check.  For instance, there may be a cancel button on the page where you are adding a new address.  A cancel button cannot be tested with an API test; therefore, you should add in a UI test for it.  Another example would be an error message that appears to the user; you may want to add in a test where you try to add an invalid address, and verify that the correct error message is displayed.  

Once you have tests that verify all of the important elements on your page, you can stop writing UI tests.  It’s not necessary to create lots of scenarios where each field is validated for various incorrect entries, because a) you already created those scenarios in your API tests, and b) you already have one UI test that verifies that the error message is displayed.  

If you already have an automated suite of UI tests, it may be a good idea to take a look at your tests and see which scenarios could be covered by API testing.  Converting your UI tests to API tests will make your regression suites faster and more reliable!

What the Sinking of the Vasa can Teach Us About Software Development

In Stockholm, Sweden, there is a museum that displays the ship called the Vasa, which sank on its maiden voyage in 1628.  I’ve never been there, but I’ve heard that the museum is fascinating for both architectural and historical reasons.  The Vasa took three years to build, and was supposed to be the flagship for Sweden’s growing navy.  The ship was built with 72 guns on two decks, and was adorned with elaborately painted carvings to reflect its majesty. 

On the day of its maiden voyage, in full view of thousands of people, including ambassadors from other countries, the ship sailed only 1400 yards before tilting, capsizing, and sinking.  It was a calm day, but a simple gust of wind caused the ship to list too much to one side, and water began pouring in through the gunports.  The primary reason for the loss of the Vasa was the simple fact that the ship’s center of gravity was too high.  How did this crucial error happen?  The answers can be helpful to us even 400 years later!

Make sure you have solid, updated plans

The shipwright in charge of building the Vasa became seriously ill (and eventually died) in the beginning stages of the project.  His assistant was in charge of completing the project, which had changed significantly since its inception.  After the initial plans were drawn, the number of guns it was expected to carry doubled, and the length of the ship was increased from 111 feet to 135 feet.  Yet the shipwright’s assistant never created a new set of plans to account for these changes.
Our lesson today: Working in an agile environment means the specifications for our software projects will frequently change.  We need to be mindful of this, and remember to re-evaluate our plans and communicate them clearly to the entire team. 

Communicate with other teams

Archeologists who have studied the Vasa and the remains of the wreckage discovered that one team of builders was using Swedish rulers, which had the modern-day 12 inches in a foot, while another team was using Amsterdam rulers, which only had 11 inches in a foot.  This resulted in the ship’s mass being distributed unevenly, compounding the problem of the high center of gravity.
Our lesson today: Most of us don’t enjoy having meetings and writing documentation, but they can be crucial in making sure that we are all on the same page.  We don’t want to waste time accidentally duplicating another team’s work, or using the wrong version of our tools.

Pay attention to your test results

Shortly before the Vasa’s first and final voyage, the captain supervising construction of the ship arranged for a demonstration of the ship’s stability.  He had thirty men run back and forth across the deck.  He stopped the test after the men had crossed the deck just three times, because the ship was rocking so much he feared it would capsize!  Rather than conduct further tests, plans continued for the launch. 
Our lesson today: Test results that don’t show us what we want to see can be disheartening, but to see a software release launch and fail feels even worse!  It’s important that testers keep digging when we see results that are different from what we expected, and it’s important that we listen to what our testers are telling us, even when it’s bad news. 

Learning about the Vasa made me marvel at just how much engineering principles have remained the same over hundreds of years.  Even though our projects are built from code rather than timber, the fundamental principles of having solid plans, communicating with everyone in the project, and getting valuable feedback through testing are still crucial to creating a great product. 

What "Passengers" Can Teach Us About Quality Assurance

Last weekend, I watched the movie Passengers. The basic plot of the movie is that two passengers in hibernation on a flight from Earth to another planet are awakened ninety years too early. As a QA engineer the movie got me thinking about two valuable lessons for developing and testing software.

Lesson One: “And Yet It Did”
In Passengers, when Jim’s hibernation pod fails, he tells the ship’s computer, the android bartender, and even another human what has happened. The response of all three is “Well, that’s impossible. The hibernation pods never fail.” Jim’s response is “Then how do you explain the fact that I’m here?” Many times in my testing career I have been told by developers that the behavior I am observing in our software is impossible. And I always respond with “And yet, here is the behavior that I’m seeing”. In one particular instance at a previous company, I was testing that information entered into the third-party software we integrated with was making it into our software. This testing was going well, until one entry didn’t travel to our software. I told the developer about it. He said, “That’s impossible. I’ve tested this, and you’ve been testing this for days.” I said, “Yes, and yet, this record wasn’t saved.” He said, “Look at the logs- you can see that the information was sent.” I said, “Yes, and yet, it wasn’t saved.” He said, “I entered more information just now, and you can see that it was saved.” I said, “Yes, and yet, the information I entered was not saved.” After much investigation, it was finally discovered that there was a bug where the database was not saving any record after the 199th record. Because I was testing in a different table than he was, and he didn’t have as many records, he didn’t see the error. The moral of the story: Even if something is impossible, it might still happen.

Lesson Two: “But What If It Did?”
One of the scariest parts of Passengers for me was that there was no way for Jim to reboot his hibernation pod and return to hibernation. Also, there were no spare pods. Even worse, there was no way for him to wake up the captain or any human who could help him. I found myself yelling at the screen, “How is this even possible? Why didn’t they put in contingency plans?” The answer, of course, is that the designers of the system were SO SURE that nothing could ever go wrong with the pods. But something did go wrong, and due to their false confidence there was no way to make it right. QA engineers are always thinking about all the possible ways that software can fail. I have often heard the argument “But no sane user would do that.” And I always respond with “But what if they did?” While we may not have time to account for every possible way that our software might fail, we should create appropriate responses for as many ways as we can, and log the rest for future fixes.

I like to think that somewhere on Earth in the Passengers universe, a QA engineer is saying to her product owners at the spaceship design company, “See, I TOLD you the pods could fail!”

Ask Your Way to Success

Ten years ago, I didn’t know how to use a Windows computer.  I didn’t know how the file system worked.  I didn’t know what right-clicking on a mouse did.  Today I am a QA Engineer doing both manual and automated testing at a great company.  How did I get here from there?

I asked a lot of stupid questions.

Most people are reluctant to ask questions, because they are afraid to look ignorant.  But I maintain that the best way to learn anything quickly is to ask questions when you don’t understand what’s going on.
Here are six ways that asking questions improves your knowledge and the health of your company:

1. Questions give others an opportunity to help you, which helps them get to know you better and establishes a rapport.  At my first official QA job, I was working with hotshot developers, all of whom were at least a decade and a half younger than me.  It was embarrassing having to admit that I didn’t know how to reset a frozen iPhone or find the shared drive in File Explorer, but I asked those questions anyway, I remembered the answers, and I showed my co-workers that I was a fast learner.

2. Questions help developers discover things they may have missed.  On countless occasions where a developer has been demonstrating a feature to me I’ll ask a question like “But what if there are no records for that user?”, or “What if GPS isn’t on?”, and they will suddenly realize that there is a use case they haven’t handled.

3. Questions keep everyone honest.  I have worked with other QA engineers who bandy about terms like “back-end call” or “a different code path” without actually knowing what they are talking about.  Asking what they mean by what they are saying makes sure that they do the work to find out what they are actually testing.  And when they get their answers, I get my answers as well.

4. Questions give you an opportunity to clear things up in your head.  You may have heard the expression “Rubber duck debugging”; I think this method works well when you’re asking questions.  I have found that sometimes just formulating the question out loud while I’m asking it clears things up for me.

5. Questions clarify expectations.  Yes, sometimes I have felt dumb saying things like “You want me to test this on the latest build, right?”, but every now and then I discover that there’s been a miscommunication, and I’d much rather find out about it before I start testing rather than after I’ve been testing the wrong thing for an hour.

6. Questions clarify priorities.  There have been many times where I’ve asked “Why are we adding this feature?”  There is almost always a good reason, but the discussion helps the team understand what the business use case is, which helps the developers decide how to design their solution.

A caveat: Don’t ask questions that you can find the answers to by using a search engine (example: “How do I find the UDID of a device using iTunes?”) or by going back and reading your email (example: “What day did we decide on for code freeze?”).  Asking these types of questions results in wasted time for everyone!

In summary, asking might make you feel silly in the short run, but it will make you and your team much smarter in the long run.  And hopefully it will create an atmosphere in which others feel comfortable asking questions as well, improving teamwork for everyone!

Fix All the Things

It’s very easy when you are rushing to complete features to let some bugs slide.  This article will show why in most cases it’s better to fix all the bugs now rather than later.  The following scenario is hypothetical, but is based on my experience as a tester.  

NewTech Inc. is very excited about offering a new email editor to their customers.  Customers will be able to compose emails to their clients and schedule when they should be sent from within the NewTech app.  NewTech’s service reps will also have the ability to add or change a company logo for the customers.  

Because the feature is on a deadline, developers are rushing to complete the work.  The QA engineer finds an issue: when changing the company logo, the logo doesn’t appear to have changed unless the user logs out and back in again.  The team discusses this issue and decides that because customers won’t see the issue (since it’s functionality that only NewTech employees can use), it’s safe to let this issue go on the backlog to be fixed at another time.

The feature is released, and customers begin using it.  The customers would all like to add their company logo to their emails, so they begin calling the NewTech service reps asking for this service.  The service reps add the logo and save, but they don’t see the logo appear on the email config page.  The dev team has forgotten to let them know that there’s a bug here, and that the workaround is to log out and back in again.  

Let’s assume that each time a service provider encounters the issue and emails someone on the dev team about it, five minutes is wasted.  If there are ten service reps on the team, that’s fifty wasted minutes.

Total time wasted to date: Fifty minutes

But now everyone knows about the issue, so it won’t be a problem anymore, right?  Wrong!  Because NewTech has hired two new QA engineers.  Neither one of them knows about the issue.  They encounter it in their testing, and ask the original QA engineer about it.  “Oh, that’s a known issue,” he replies.  “It’s on the backlog.”  Time wasted: ten minutes for each new QA engineer to investigate the problem, and ten minutes for each new engineer to ask the first QA engineer about it.

Total time wasted to date:  One hour and ten minutes

Next, a couple of new service reps are hired.  At some point, they each get a request to from a company to change the company’s logo.  When they go to make the change, the logo doesn’t update.  They don’t know what’s going on, so they ask their fellow service reps.  “Oh yeah, that’s a bug,” say the senior service reps.  “You just need to log out and back in again.”  Time wasted: ten minutes for each new service provider to be confused about what’s going on, and ten more minutes in conversation with the senior service reps.

Total time wasted to date: One hour and thirty minutes

It’s time to add new features to the application.  NewTech decides to give their end users the option of adding a profile picture to their account.  A new dev is tasked with adding this functionality.  He sees that there’s an existing method to add an image to the application, so he chooses to call that method to add the profile picture.  He doesn’t know about the login/logout bug.  When one of the QA engineers tests the new feature, she finds that profile picture images don’t refresh unless she logs out and back in again.  She logs a new bug for the issue.  Investigating the problem and reporting it takes twenty minutes.

Total time wasted to date:  One hour and fifty minutes

The dev team meets and decides that because customers will see the issue, it’s worth fixing.  The dev who is assigned to fix the issue is a different dev from the one who wrote the image-adding method (who has since moved on to another company), so it takes her a while to familiarize herself with the code.  Time spent fixing the issue: two hours.  

Total time wasted to date: Three hours and fifty minutes

The dev who fixed the issue didn’t realize that the bug existed for the company logo as well, so didn’t mention it to the QA engineer assigned to test her bug fix.  The QA engineer tests the bug fix and finds that it works correctly, so she closes the issue.  Time spent testing the fix: thirty minutes.

Total time wasted to date: Four hours and twenty minutes

When it’s time for the new feature to be released, the QA team does regression testing.  They discover that there is now a new issue on the email page: because of the fix for the profile images, now the email page refreshes when customers make edits, and the company logo disappears from the page.  One of the QA engineers logs a separate bug for this issue.  Time spent investigating the problem and logging the issue: thirty minutes.

Total time wasted to date: Four hours and fifty minutes

The dev team realizes that this is an issue that will have significant impact on customers, so the developer quickly starts working on a fix.  Now she realizes that the code she is working on affects both the profile page and the email page, so she spends time checking her fix on both pages.  She advises the QA team to be sure test both pages as well.  Time spent fixing the problem: one hour.  Time spent testing the fixes: one hour.

Total time wasted to date: Six hours and fifty minutes

How much time would it have taken for the original developer to fix the original issue?  Let’s say thirty minutes, because he was already working with the code.  How much time would it have taken to test the fix?  Probably thirty minutes, because the QA engineer was already testing that page, and the code was not used elsewhere.  

So, by fixing the original issue when it was found, NewTech would have saved nearly six hours in work that could have been spent on other things.  This doesn’t seem like a lot, but when considering the number of features in an application, it really adds up.  And this scenario doesn’t account for lost productivity from interruptions.  If a developer is fielding questions from the service reps all day about known issues that weren’t fixed because they weren’t customer-facing bugs, it’s hard for him to stay focused on the coding he’s doing.

The moral of the story is: unless you think that no user, internal or external, will ever encounter the issue, fix things when you find them!

How to Train Your Dev

Training your Dev is really about training yourself.  A more accurate (but much less catchy) title for this blog post would be “How to work and communicate effectively in order to facilitate a productive relationship with developers”.  

There are two steps to having a good working relationship with your dev: 1) developing good work habits, and 2) communicating clearly.  We’ll take a look at these two steps in detail.

Good Work Habits:

• Make sure you have completely read a feature’s Acceptance Criteria and all available notes and documents.  This can help prevent unnecessary and time-consuming misunderstandings.
• Ask questions if there is anything in the feature that you don’t understand.  Don’t make potentially incorrect assumptions.
• Document your work.  This is especially helpful when you have found an issue and the developer needs to know what browser you were using or what server you are pointing to.
• Check twice to make sure that you really have a bug.  Perhaps what you are seeing is a configuration problem, a connection problem, or simply user error.

Communicating Clearly:

• Learn the preferred communication style of your dev and use it.  For example, some developers like to hear about issues immediately, and testing and bug fixing become a collaboration.  Other developers prefer to hear right away only if the issues are big ones, and would rather have you document the smaller issues for a later conversation.
• Ask your dev to walk you through any confusing features.  He or she will be happy to explain things to you, because they know that any information they give you at the outset of testing will save misunderstandings later.
• Be kind when reporting issues.  Your dev has worked hard on the feature he or she has delivered, and we all know it’s no fun to have our work criticized.
• Give feedback in the form of a question.  This can soften the blow of finding a bug.  For example, “I noticed that when I clicked the Save button, I wasn’t taken to the next page.  Is this as designed?”
• Let your dev know what he or she can do to help you do your job more efficiently.  A good example of this is asking them to not assign an issue to you until it is actually in the test environment, so you won’t inadvertently start testing it before the code is there.

A good working relationship with your dev is all about trust!  You trust that your dev has completed the work they’ve assigned to you, that they’ve done some of their own testing before the handoff, that the work is in the QA environment and ready for testing, and that your dev has let you know about any potential areas of regression to test.  

In turn, your developer trusts that you have tested everything in the Acceptance Criteria, that you’ve done regression testing, that you’ve tested with various security levels and on various browsers, that the issues you’ve found are legitimate, and that you will clearly communicate what you tested and what issues you found.  

Train yourself to work effectively and communicate clearly, and you will find this level of trust in your relationship with all the developers you work with!

The Trouble With Toggles

Some of you may have seen the classic Star Trek episode, "The Trouble with Tribbles".  In the episode, Lt. Uhura gets a gift of a cute little ball of fur from a passing trader.  It's a huge hit with the crew, and when the tribble has babies, everyone who wants one can enjoy these little fuzzy creatures.  The trouble begins when they discover that the tribbles are born pregnant and quickly give birth, increasing the tribble population exponentially until they are taking over the entire ship. 

Today I would like to talk about the trouble with toggles, and what they have in common with tribbles.  Toggles can be helpful to developers in many ways.  They make it easier for development of a new feature to continue in the main branch, without disturbing the work of others.  They enable us to do beta testing, where a handful of customers try out a feature or layout and provide their feedback while most customers continue using the software without the changes.  And they enable us to quickly remove a feature if it's discovered that it has a critical error.  But toggles can be problematic for two very important reasons, which I will explain using our tribble friends.

First, toggles make more work for QA engineers.  Just as the tribbles multiplied exponentially, the number of test passes we need to do when using toggles also multiplies exponentially.  Consider the following scenario: Team One adds a new feature with a toggle.  In order to make sure that the feature hasn't broken any existing functionality, they need to do two passes of smoke tests: one with the toggle on, and one with the toggle off.  Next they add a second feature with a toggle.  Now there are four different combinations of scenarios that their software can have: both toggles on, toggle A on and toggle B off, toggle B on and toggle A off, and both toggles off.  So they'll need to do four different passes of smoke tests to make sure that all their features are working correctly.  Now let's add in Team Two.  They have two toggles as well, so they'll need to do four passes of smoke tests for all the permutations of toggle A and toggle B. 

As release day approaches, Team One and Team Two are merging their code into the main branch.   Now there are EIGHT different possible combinations for toggle states. Even if the teams decide that it's a low risk to skip smoke testing in all eight scenarios, there will still be a lot of investigative work involved for every bug found.  Let's say someone finds a bug in an instance where Toggle A and C are on and Toggle B and D are off.  They'll need to answer questions like "What if just Toggle C is on?  Do we still see the bug? What about if Toggle D is turned on as well?  Does the bug go away?"  Research like this can be really time-consuming.  This is time that could be better spent doing exploratory testing.  If all the features were enabled in all scenarios, testers would only need to do one pass of smoke tests and spend the rest of their time digging deeper into each new feature.

Secondly, toggles can give software teams a false sense of security, which can lead to buggy software and tech debt.  In "The Trouble with Tribbles", the tribbles' purring sound had a soothing effect on human's nervous systems, which made the crew pay less attention to the fact that in a couple of days the tribble population would fill the Enterprise from deck to ceiling.  Similarly, engineers can be soothed by the knowledge that they can always turn the feature off, which means that they can test the feature less carefully. If bugs are found, they can be dismissed with the statement, "Well, it's Beta, and the Beta users will understand.  We'll log it and take a look at it later." Bugs like these can compound, especially when mixed with all the other bugs in all the other Beta features.  And the bugs can take a lot longer to fix six months later when the original developers don't remember what they did or may not even be with the company any more. Alternatively, if the teams tested every feature with the knowledge that all users would be seeing it on first release, they would test more carefully and fix bugs right away, ensuring much greater code quality from the outset.

Like tribbles, toggles can be fun, in that they enable us to add more pizazz more quickly to an application.  But as the crew of the Enterprise found, too much of a good thing has its consequences!