We are implementing a new QA process to improve product quality and development efficiency.

I. QA Process:

1. Before each release we decide:

   1. Full QA OR QA-ing only the most important features + QA-ing the new feature. Depending on what will released, a QA “type” will be decided.

   2. Who will do the QA

      To enhance ownership, preferably 70% of the time should the same persons) however all team members are highly encouraged to participate in a full QA process at least once to:

      1. Have the opportunity to take the user’s role and use the platform.
      2. Have an idea about how much time and effort the manual tests are taking.

   🟢 The most important features:

   1. Create an LLM app from template (single prompt and chat-app)
   2. Create an LLM app from code
   3. Access an old LLM app
   4. In each LLM app 1. new from code, 2. new from template, 3. old from template 4. old from code we are able to:
      1. Run a prompt from the playground
      2. Run an evaluation and see results
      3. See corresponding data in observability

II. Goals

Q3’s goal : Increasing Test Coverage by 70%

In order to achieve this goal we suggest

• Team members should write tests for new features or bug fixes.
• Adding automated tests is now part of the Definition of Done.

III. Roadmap:

[Still an Experiment] Adopting Behavior-Driven Development (BDD) and Gherkin :

We're implementing BDD along with Gherkin syntax to enhance our development process. This approach offers several advantages:

1. Standardized Format: Gherkin provides a clear, structured way to define features, superior to spreadsheets or unstructured documents.
2. Improved Collaboration: BDD bridges the gap between technical and non-technical team members, fostering better communication and shared understanding of requirements.
3. Living Documentation: Features written in Gherkin serve as both specifications and tests, keeping our documentation always up-to-date.
4. Easier Test Management: The structured nature of Gherkin facilitates easier integration with and migration between test management systems.
5. Focus on User Value: BDD encourages thinking from the user's perspective, helping us create more relevant and valuable features.

Centralized Feature Definition:

We will establish a centralized location to declare all features (example of declaring features). This ensures consistency and provides a single source of truth for our feature definitions. The exact location is still under consideration, with two main options:

1. Within cloud/oss repositories
2. In a separate, dedicated repository

Testomat Integration:

A process will be implemented to sync Gherkin-notated features with Testomat in order to have always all the features.