Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions src/content/docs/project/runs/running-automated-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,15 +118,15 @@ Testomat.io reporter can be configured to add additional information for Run rep

When you enable reporing for tests running in parallel, you might end with multiple reports per each executed process. There are few options to deal with this case, which you can use depending on your setup.

**Strategy 1: Use start-test-run**
**Strategy 1: Use @testomatio/reporter run**

Run tests via `npx start-test-run` command:
Run tests via `npx @testomatio/reporter run` command:

```
npx start-test-run -c "<actual run command>"
npx @testomatio/reporter run "<actual run command>"
```

Under hood start-test-run creates a new empty run and passes its ID as environment variable into all spawned processes. So no matter how many parallel processes are started they will report to the single Run report.
Under hood `@testomatio/reporter run` creates a new empty run and passes its ID as environment variable into all spawned processes. So no matter how many parallel processes are started they will report to the single Run report.

![Alt text](./images/image-10.png)

Expand Down Expand Up @@ -166,10 +166,10 @@ In this case you create a run, receive its ID and manually close it after all ru

![Alt text](./images/image-9.png)

Create a run via `start-test-run --launch`:
Create a run via `@testomatio/reporter start`:

```
export TESTOMATIO_RUN=$(TESTOMATIO=xxx npx start-test-run --launch | tail -1)
export TESTOMATIO_RUN=$(TESTOMATIO=xxx npx @testomatio/reporter start | tail -1)
```

then execute tests passing the `TESTOAMTIO_PROCEED` variable:
Expand All @@ -178,10 +178,10 @@ then execute tests passing the `TESTOAMTIO_PROCEED` variable:
TESTOMATIO=xxx TESTOMATIO_RUN=$RUN_ID TESTOMATIO_PROCEED=1 <actual-run-command>
```

Once tests are finished close the run with `start-test-run --finish`:
Once tests are finished close the run with `@testomatio/reporter finish`:

```
TESTOMATIO=xxx npx start-test-run --finish
TESTOMATIO=xxx npx @testomatio/reporter finish
```

If you have a complex pipeline, you can start Run on the stage #1, execute tests in parallel on stage #2, and close the run on stage #3.
Expand Down
22 changes: 11 additions & 11 deletions src/content/docs/test-reporting/frameworks.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,17 +55,17 @@ TESTOMATIO={API_KEY} npx codeceptjs run

#### CodeceptJS Parallel Run

If tests run parallel, like workers in CodeceptJS use `start-test-run` command to get proper reports:
If tests run parallel, like workers in CodeceptJS use `@testomatio/reporter run` command to get proper reports:

```bash
TESTOMATIO={API_KEY} npx start-test-run -c 'npx codeceptjs run-workers 2'
TESTOMATIO={API_KEY} npx @testomatio/reporter run 'npx codeceptjs run-workers 2'
```

> Specify a command to run with `-c` option in `start-test-run`
> Specify a command to run as argument to `@testomatio/reporter run`

Use `--env-file <envfile>` option to load environment variables from .env file. Inside env file TESTOMATIO credentials like `TESTOMATIO` api key or [S3 config for artifacts](./artifacts.md).

Command `start-test-run` is used to initiate a single run report before all workers are started. Each worker will report to the same Run, and after all workers and codeceptjs finishes, this will finish the run report.
Command `@testomatio/reporter run` is used to initiate a single run report before all workers are started. Each worker will report to the same Run, and after all workers and codeceptjs finishes, this will finish the run report.

> 📑 [Example Project](https://github.com/testomatio/examples/tree/master/codeceptJS) | 🗄 [CodeceptJS API Example](https://github.com/testomatio/examples/tree/master/codeceptJSApi) | 🥒 [CodeceptJS Cucumber Example](https://github.com/testomatio/examples/tree/master/codeceptjs-cucumber)

Expand Down Expand Up @@ -257,7 +257,7 @@ For making screenshots on failed tests add the following hook to `wdio.conf.js`:
Run the following command from you project folder:

```bash
TESTOMATIO={API_KEY} npx start-test-run -c 'npx wdio wdio.conf.js'
TESTOMATIO={API_KEY} npx @testomatio/reporter run 'npx wdio wdio.conf.js'
```

> 📑 [Example Project](https://github.com/testomatio/examples/tree/master/webdriverio-mocha)
Expand Down Expand Up @@ -342,7 +342,7 @@ exports.config = {
Run the following command from you project folder:

```bash
TESTOMATIO={API_KEY} npx start-test-run -c 'npx protractor conf.js'
TESTOMATIO={API_KEY} npx @testomatio/reporter run 'npx protractor conf.js'
```

> 📑 [Example Project](https://github.com/testomatio/examples/tree/master/protractor)
Expand Down Expand Up @@ -383,7 +383,7 @@ In this case JUnit XML will be saved into `target/surefire-reports/`
So you can import reports to Testomat.io by running:

```
TESTOMATIO={API_KEY} npx report-xml "target/surefire-reports/**.xml" --java-tests
TESTOMATIO={API_KEY} npx @testomatio/reporter xml "target/surefire-reports/**.xml" --java-tests
```

> **Note**
Expand All @@ -405,10 +405,10 @@ In this example, we added ID as a comment to `negativeNumbersCanBeSubtracted` te
}
```

To make this feature work, please ensure that source code of Java tests is accessible to `npx report-xml` command, use `--java-tests` option to specify the correct path. To check if source code of tests is available run reporter with DEBUG mode:
To make this feature work, please ensure that source code of Java tests is accessible to `npx @testomatio/reporter xml` command, use `--java-tests` option to specify the correct path. To check if source code of tests is available run reporter with DEBUG mode:

```
DEBUG=@testomatio/reporter:* TESTOMATIO={API_KEY} npx report-xml "target/surefire-reports/**.xml" --java-tests
DEBUG=@testomatio/reporter:* TESTOMATIO={API_KEY} npx @testomatio/reporter xml "target/surefire-reports/**.xml" --java-tests
```

Is a source code is not available, test IDs can be set from output. To set Test ID to a test, copy test ID and print it from a test:
Expand Down Expand Up @@ -476,10 +476,10 @@ Also we use `TESTOMATIO_TITLE_IDS=1` to write test IDs into scenario titles inst

To generate JUnit reports, you can use the built-in Cucumber JUnit plugin. When you run your Cucumber tests the JUnit reports will be generated in the default directory (`build/reports/tests/test`) in XML format.

To submit report to Testomat.io use `npx report-xml` command from `@testomatio/reporter` NodeJS package:
To submit report to Testomat.io use `npx @testomatio/reporter xml` command from `@testomatio/reporter` NodeJS package:

```
TESTOMATIO={API_KEY} npx report-xml "build/reports/tests/**/*.xml"
TESTOMATIO={API_KEY} npx @testomatio/reporter xml "build/reports/tests/**/*.xml"
```

If you want to have artifacts attached, use `System.out.println` to print an absulute path to file that should be uploaded as a screenshot.
Expand Down
12 changes: 6 additions & 6 deletions src/content/docs/test-reporting/junit.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ Run your test framework and generate a JUnit report.
Then import XML report into Testomat.io

```
npx report-xml "{pattern}" --lang={lang}
npx @testomatio/reporter xml "{pattern}" --lang={lang}
```

- `pattern` - is a glob pattern to match all XML files from report. For instance, `"test/report/**.xml"` or just `report.xml`
Expand All @@ -69,7 +69,7 @@ pytest --junit-xml report.xml
Import report with this command

```
TESTOMATIO={API_KEY} npx report-xml report.xml --lang=python
TESTOMATIO={API_KEY} npx @testomatio/reporter xml report.xml --lang=python
```

## JUnit
Expand All @@ -83,7 +83,7 @@ mvn clean test
Import report with this command:

```
TESTOMATIO={API_KEY} npx report-xml "target/surefire-reports/*.xml" --java-tests
TESTOMATIO={API_KEY} npx @testomatio/reporter xml "target/surefire-reports/*.xml" --java-tests
```

> [!NOTE]
Expand Down Expand Up @@ -157,7 +157,7 @@ public void testAddition() {
Generate NUnit XML report and run the following code:

```
TESTOMATIO={API_KEY} npx report-xml "report.xml" --lang="c#"
TESTOMATIO={API_KEY} npx @testomatio/reporter xml "report.xml" --lang="c#"
```

If NUnit generates `<ResultFiles>` section in XML report, all items from it will be uploaded as artifacts. For instance, this is available for **Playwright DotNet** integration.
Expand Down Expand Up @@ -209,7 +209,7 @@ rails test
Import reports from `test/reports` directory:

```
TESTOMATIO={API_KEY} npx report-xml "test/reports/*.xml" --lang ruby
TESTOMATIO={API_KEY} npx @testomatio/reporter xml "test/reports/*.xml" --lang ruby
```

### Assign Test ID
Expand Down Expand Up @@ -243,6 +243,6 @@ Use `tid://` prefix with a existing Test ID to match test with ID.
Generate PHPUnit XML and import it

```
TESTOMATIO={API_KEY} npx report-xml "report.xml" --lang php
TESTOMATIO={API_KEY} npx @testomatio/reporter xml "report.xml" --lang php
```

26 changes: 13 additions & 13 deletions src/content/docs/test-reporting/pipes/testomatio.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ TESTOMATIO={API_KEY} TESTOMATIO_PROCEED=1 TESTOMATIO_RUN={RUN_ID} <actual run co
After all reports were attached and run can be execute the following command:

```
TESTOMATIO={API_KEY} TESTOMATIO_RUN={RUN_ID} npx start-test-run --finish
TESTOMATIO={API_KEY} TESTOMATIO_RUN={RUN_ID} npx @testomatio/reporter finish
```

### Setting Report Title
Expand Down Expand Up @@ -116,7 +116,7 @@ TESTOMATIO={API_KEY} TESTOMATIO_ENV="Windows, Chrome" <actual run command>
If you want to create a run and obtain its `{RUN_ID}` from [testomat.io](https://testomat.io) you can use `--launch` option:

```bash
TESTOMATIO={API_KEY} npx start-test-run --launch
TESTOMATIO={API_KEY} npx @testomatio/reporter start
```

This command will return `{RUN_ID}` which you can pass to other jobs in a workflow.
Expand All @@ -128,7 +128,7 @@ This command will return `{RUN_ID}` which you can pass to other jobs in a workfl
If you want to finish a run started by `--launch` use `--finish` option. `TESTOMATIO_RUN` environment variable is required:

```bash
TESTOMATIO={API_KEY} TESTOMATIO_RUN={RUN_ID} npx start-test-run --finish
TESTOMATIO={API_KEY} TESTOMATIO_RUN={RUN_ID} npx @testomatio/reporter finish
```

### Setting Build URL
Expand Down Expand Up @@ -199,7 +199,7 @@ Testomatio allows you to enhance your test execution with the option to filter a
To use the `--filter` option with Testomatio, follow the format:

```bash
TESTOMATIO={API_KEY} npx start-test-run -c <actual run command> --filter 'testomatio:{filter-type}={filter-value}'
TESTOMATIO={API_KEY} npx @testomatio/reporter run <actual run command> --filter 'testomatio:{filter-type}={filter-value}'
```

#### Available Filter Types
Expand All @@ -213,7 +213,7 @@ Filter tests by tag names. Use this filter to select tests associated with speci
_Command execution example_

```bash
TESTOMATIO={API_KEY} npx start-test-run -c 'actual run command' --filter 'testomatio:tag-name=smoke'
TESTOMATIO={API_KEY} npx @testomatio/reporter run 'actual run command' --filter 'testomatio:tag-name=smoke'
```

**Plan ID Filter**
Expand All @@ -225,7 +225,7 @@ Filter tests by Testomatio plan IDs. Use this filter to organize and execute tes
_Command execution example_

```bash
TESTOMATIO={API_KEY} npx start-test-run -c 'actual run command' --filter 'testomatio:plan-id=b62f3170'
TESTOMATIO={API_KEY} npx @testomatio/reporter run 'actual run command' --filter 'testomatio:plan-id=b62f3170'
```

**Label Filter**
Expand All @@ -244,13 +244,13 @@ Another Label Filter Example 3:
_Command execution example_

```bash
TESTOMATIO={API_KEY} npx start-test-run -c 'actual run command' --filter 'testomatio:label=new-label-test'
TESTOMATIO={API_KEY} npx @testomatio/reporter run 'actual run command' --filter 'testomatio:label=new-label-test'
```

OR

```bash
TESTOMATIO={API_KEY} npx start-test-run -c 'actual run command' --filter 'testomatio:label=NEW label TEST'
TESTOMATIO={API_KEY} npx @testomatio/reporter run 'actual run command' --filter 'testomatio:label=NEW label TEST'
```

**Label Filter based on the Severity type**
Expand All @@ -272,13 +272,13 @@ Label Filter Example based on the Severity type:
_Command execution example_

```bash
TESTOMATIO={API_KEY} npx start-test-run -c 'actual run command' --filter 'testomatio:label=severity-f1b11:🔥 Major'
TESTOMATIO={API_KEY} npx @testomatio/reporter run 'actual run command' --filter 'testomatio:label=severity-f1b11:🔥 Major'
```

OR

```bash
TESTOMATIO={API_KEY} npx start-test-run -c 'actual run command' --filter 'testomatio:label=NEW label TEST:💤 Trivial'
TESTOMATIO={API_KEY} npx @testomatio/reporter run 'actual run command' --filter 'testomatio:label=NEW label TEST:💤 Trivial'
```

**JIRA Ticket Filter**
Expand All @@ -290,21 +290,21 @@ Filter tests linked to JIRA tickets. Use this filter to coordinate testing effor
_Command execution example_

```bash
TESTOMATIO={API_KEY} npx start-test-run -c 'actual run command' --filter 'testomatio:jira-ticket=LMP-51'
TESTOMATIO={API_KEY} npx @testomatio/reporter run 'actual run command' --filter 'testomatio:jira-ticket=LMP-51'
```

**📚 Example Usage**

For instance, if you want to run tests with a specific label named "new-label-test," you can use the following command:

```bash
TESTOMATIO={API_KEY} npx start-test-run -c 'npx codeceptjs run' --filter 'testomatio:label=new-label'
TESTOMATIO={API_KEY} npx @testomatio/reporter run 'npx codeceptjs run' --filter 'testomatio:label=new-label'
```

OR using Playwright:

```bash
TESTOMATIO={API_KEY} npx start-test-run -c 'npx playwright test' --filter 'testomatio:label=new-label'
TESTOMATIO={API_KEY} npx @testomatio/reporter run 'npx playwright test' --filter 'testomatio:label=new-label'
```

> Remember to replace {API_KEY} with your actual Testomatio API key and adjust the `<actual run command>` according to your testing setup
Expand Down
6 changes: 3 additions & 3 deletions src/content/docs/test-reporting/workflows.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ jobs:
with:
node-version: 16.x
- run: npm i
- run: npx start-test-run -c 'npx codeceptjs run-workers 2 --grep "${{ github.event.inputs.grep }}"'
- run: npx @testomatio/reporter run 'npx codeceptjs run-workers 2 --grep "${{ github.event.inputs.grep }}"'
env:
TESTOMATIO: '${{ github.event.inputs.testomatio }}'
TESTOMATIO_RUN: '${{ github.event.inputs.run }}'
Expand Down Expand Up @@ -109,7 +109,7 @@ jobs:
env:
RAILS_ENV: test
- name: Testomatio Report
run: npx report-xml "test/reports/**.xml" --lang=Ruby
run: npx @testomatio/reporter xml "test/reports/**.xml" --lang=Ruby
if: always()
env:
TESTOMATIO: ${{ secrets.TESTOMATIO }}
Expand Down Expand Up @@ -144,7 +144,7 @@ jobs:
run: vendor/bin/phpunit --log-junit=report.xml

- name: Testomatio Report
run: npx report-xml report.xml --lang=PHP
run: npx @testomatio/reporter xml report.xml --lang=PHP
if: always()
env:
TESTOMATIO: ${{ secrets.TESTOMATIO }}
Expand Down