Skip to main content

Quickstart

Testiny integrates with test automation tools and frameworks to collect and manage automated test results in one place. Whether tests are run via a CI/CD pipeline or manually, Testiny helps debug, troubleshoot, and analyze failures efficiently. Learn how to submit your first automated test run to Testiny and set up reporting with your automation framework and CI/CD pipeline.

Watch a short overview of how automated test results look in Testiny:

Submit your first automation results​

  1. Download or install the Testiny CLI and create an API key in Testiny. Pass the key to the CLI via the TESTINY_API_KEY environment variable.

  2. Run your automated tests & generate a report. Testiny CLI supports JUnit, xUnit, NUnit, TestNG, Playwright, Cypress, Mochawesome, CodeceptJS, Vitest, VSTest (trx), Ranorex reports & more. See the CLI reference for all supported report formats and their options.

  3. Import your automated results to Testiny. Specify the Testiny project ID, a source name, the report type and the report file(s):

export TESTINY_API_KEY=****
npx @testiny/cli automation --project ID_OR_KEY --source "unit-tests" --junit report.xml
note

If you are using Testiny Server (self-hosted), add the --app option with the URL of your Testiny instance, for example --app https://testiny.example.com.

See command output
API key: user '[email protected]' in organization 'testiny'
Using project: Demo (DE)
Import source: unit-tests
Environment: gitlab
Processing junit file: report.xml
  Submitted 168/168 results
Completed 1 automation test run(s)
Imported all 1 file(s) successfully
Submitted Testiny automated run ids: 264

The CLI parses the report file, creates an automation run in the specified project and adds all results, errors and attachments to that run.

The source name groups your test runs together, so choose a name that fits your test suite — e.g. frontend-tests, backend-tests, e2e-tests, api-tests. If the CLI runs in a CI pipeline, it also auto-detects the environment and attaches details such as the commit hash and pipeline ID to the results.

The console output shows how many files were processed, how many results and attachments were uploaded, and the id of the created run. Open that run in Testiny to see your results — it appears in the project you specified, grouped under your source.

Watch this short demo to see how importing automated test results works in Testiny:

Run Fields​

Run Fields (and the specified source) are used to identify test runs and to group together automated results. Testiny compares the values of these fields to decide whether incoming results are added to an existing open run or start a new one. In CI environments, the Pipeline ID, Build ID, Job ID or any combination of them make good Run Fields.

If you don't set --run-fields, the CLI picks the defaults for you: in a known CI environment it uses that environment's default Run Fields (for example ci_pipeline_id in GitLab pipelines), and outside a CI environment it generates a run identifier.

To define your own Run Fields, combine two options:

  • Optional: use --field-values to add a field and its value to every result
  • use any field(s) from your automated results as a Run Field and pass them as a comma-separated list with --run-fields
export TESTINY_API_KEY=****
npx @testiny/cli automation --project ID_OR_KEY --source "unit-tests" --field-values my_run_key=1234 --run-fields my_run_key --junit report.xml

To find out which field names are available in your environment, run the automation command with the --show-env option. Our CI/CD examples also list the collected fields and the default Run Fields for each supported environment. The same field values can be used to build the run title with --run-title-pattern.

After submitting results to Testiny, the test run is automatically marked as completed and no new results can be added to this run. If you would like to submit more results to the same run, add the --incomplete flag when submitting results. See the section Parallel and Sharded Test Runs for an example.

Parallel and Sharded Test Runs​

Run Fields are especially useful when importing results from split tests (e.g. parallel testing, test sharding, different environments, ...), as you can use the available environment variables to add the distributed results to the same test run.

If you would like to manually group together reports and import them to the same run in Testiny, specify your own Run Field, mark the imports as --incomplete and finally, mark the run as complete with --complete-runs:

export TESTINY_API_KEY=****

# report 1
npx @testiny/cli automation --project ID_OR_KEY --source "unit-tests" --incomplete --field-values my_run_key=1234 --run-fields my_run_key --junit report1.xml

# report 2
npx @testiny/cli automation --project ID_OR_KEY --source "unit-tests" --incomplete --field-values my_run_key=1234 --run-fields my_run_key --junit report2.xml

# complete run
npx @testiny/cli automation --project ID_OR_KEY --source "unit-tests" --complete-runs --field-values my_run_key=1234 --run-fields my_run_key

In a CI environment, the CLI auto-detects variables like commit hashes and tags, pipeline and job IDs, and adds them as fields to your automation run — so in most cases you don't need to set Run Fields yourself. See our examples for CI/CD integration for the fields and defaults of each supported environment.

Importing multiple files​

Some automation frameworks like Cypress or Mochawesome output multiple result files. You simply import multiple report files at once by specifying a directory or a pattern to select multiple files for import:

export TESTINY_API_KEY=****
npx @testiny/cli automation --project ID_OR_KEY --source unit-tests --junit results/*.xml

Test Automation Guide​

Learn more about automation concepts, adding fields to your run and parallel testing in our detailed Automation Guide.

Reporters & examples​

For some automation frameworks, we offer custom reporters where you don't need to install and handle the CLI yourself. See all available reporters and usage examples.

Next steps​

Last updated on