Introduction

• We are using BDD and Gherkin for QA’ing. But what does it mean?

  • We define our manual tests using a structured format called Gherkin (see below). We can then load this file into a tool called Testomat, which allows streamlines running manual tests (see screenshot below)

    

  • Gherkin + BDD is useful and saves us time because:

    • We don't need to determine what to test each time we do QA. We can simply add new functionality tests if needed and select existing tests from our pre-built library.
    • Our testing approach is systematic. We add new tests as we discover issues, gradually improving our product's quality over time.
    • The process is streamlined: testing and issue creation are integrated. Testomat's integration with Linear allows us to create issues while performing QA.
    • It allows us to slowly move from manual to automatic tests

What is this Gherkin syntax?

• You can find a the reference here
• To illustrate, let's consider adding two tests for observability in the playground. The first test checks if I can see the trace when running an application. The second verifies that I can add a span value from the trace to a test set. Here's how these tests would look (see comments for explanation):

Feature: Observability in Playground
	Background:
		Given that I have created a single prompt application
	
	Scenario: Seeing a trace
		When I run an application in the playground
		Then I can see a view trace link under the results
		And if I click on it, I can see the trace
		
	Scenario: Adding a trace to a test set
		When I run an application in the playground
		And I a the trace to the test set in the trace view
		Then I can see the content of the trace in a new test case in the test set
	

Our Process:

When creating the story (author of RFC/PRD+team):

• Create gherkin test suites based on the PRD / designs

When creating a PR (author):

• Before making the PR as ready to review → Run smoke test and sanity tests and acceptance tests to make sure that the basic functionality of the app works → No need to use testomat here, unless you find it helpful
• Before marking the PR as ready for QA:
  • → Add Gherkin tests or test suites for the new features' acceptance criteria
  • → Create a new Test Plan for the feature, including the Linear issue number (For simpler cases, document it in Slack or Linear, e.g., "regression tests for all features under the observability folder")
  • → Tag the PR as "for-qa"
  • → Move the issue in Linear to the QA stage
  • → Communicate in advance during Standup when it will be ready, create a Slack thread proposing someone for QA, and prepare for QA (via Standup, Slack, and determining who will perform QA)

When reviewing a PR (reviewer):

• Run acceptance tests and exploratory tests to discover wrong behavior. Add tests to gherkin if needed. → No need to use testomat here, unless you find it helpful

When QA’ing before a merge:

• Run the tests in testomat
• Create issues for problems
• Share the test report in Slack under the thread

When QA’ing before a release:

• Run the tests in testomat