Overview

What is testsuite

Testsuite is a microservice-oriented test framework written in Python based on pytest.

Testsuite is written and supported by Yandex.Taxi, and is used to test Yandex.Taxi microservices written in C++ and Python.

The principal suggested approach to testing - although not the only one - is black box, when the service is tested through http calls.

Direct read and write access from test to database is supported to enable precondition setup and result assertions.

Installation

Installation using pip:

pip3 install yandex-taxi-testsuite

# testsuite with mongodb support
pip3 install yandex-taxi-testsuite[mongodb]

# testsuite with postgresql support
pip3 install yandex-taxi-testsuite[postgresql]
pip3 install yandex-taxi-testsuite[postgresql-binary]

# testsuite with redis support
pip3 install yandex-taxi-testsuite[redis]

# testsuite with mysql support
pip3 install yandex-taxi-testsuite[mysql]

# testsuite with clickhouse support
pip3 install yandex-taxi-testsuite[clickhouse]

# testsuite with rabbitmq support
pip3 install yandex-taxi-testsuite[rabbitmq]

# testsuite with kafka support
pip3 install yandex-taxi-testsuite[kafka]

Supported databases

Out-of-the-box testsuite supports the following databases:

  • PostgreSQL

  • MongoDB

  • Redis/Valkey

  • MySQL/MariaDB 10+

  • ClickHouse

  • RabbitMQ

Supported operating systems

Testsuite runs on GNU/Linux and macOS operating systems.

Principle of operation

Testsuite sets up the environment for the service being tested:

  • Testsuite starts any required databases (postgresql, mongo, redis).

  • Before each test, testsuite fills the database with test data.

  • Testsuite starts its own web server (mockserver), which mimics (mocks) microservices other than the one being tested.

Testsuite starts the microservice being tested in a separate process.

Testsuite then runs tests.

A test performs http requests to the service and verifies that the requests were processed properly.

A test may check the results of an http call by looking directly into the service’s database.

A test may check whether the service has made calls to external services, as well as the order of calls and the data that was sent and received.

A test may check the internal state of the service as represented by the data the service sent to the test with the testpoint mechanism.

Source code

Testsuite open-source edition code is available here.

Documentation

For full documentation, including installation, tutorials, please see https://yandex.github.io/yandex-taxi-testsuite/.

Running testsuite

self-tests:

pytest3 ./tests

tests of example services:

cd docs/examples && make

Development

Setup virtual env

In order to test your modifications it’s useful to run testsuite inside virtualenv. Use the following command to create developer’s venv:

make setup-dev-venv

Virtualenv will be created in .venv-dev directory.

Code format and linters

Auto format source code:

make venv-format

Run linters:

make venv-check-linters
make venv-check-mypy

You can also add pre-commit hook which will run ruff and linters for you:

make install-pre-commit-hooks

Running tests

You can run tests using Makefile:

make venv-tests

Or directly using pytest:

make setup-dev-venv                  # Setup virtual env first
. .venv-dev/bin/activate             # Activate virtualenv
pytest -vv tests/plugins/mockserver  # Finally run pytest

Building documentation

make build-docs

Contributing

  • Fork project

  • Write code in your own branch

  • Test your code

  • Run linters and code formatters

  • Add new changelog entry

  • Create pull request on github

Changelog

0.3.3

  • Add servicelogs plugin (#186)

  • Add logcapture plugin (#184)

  • Add asyncio_socket (#184)

  • add ci tests to ensure, that userver testsuite tests works (#185)

0.3.2

  • restore testsuite/annotations.py: userver depends on it (#183)

0.3.1

  • Add experimental version of CompareTransform (#182)

  • Specify tmp dir path inside tests (#181)

  • split tests by core and databases (#180)

0.3.0

  • remove unused imports from testsuite (#175)

  • drop 3.7 and 3.8 support (#178)

  • Drop legacy loop fixutre support (#177)

  • Support external develop branch

  • Fix assertrepr_compare_experimental mapping usage (#176)

0.2.21

  • asyncio legacy: close event loop (#173)

  • Note valkey in docs (#172)

  • Add Valkey support (#169)

  • Revert “fix assertrepr_compare_experimental.py: replace pytest.Config with pytest.config.Config” (#170)

0.2.20

  • acallqueue: support callable instances (#166)

  • fix assertrepr_compare_experimental.py: replace pytest.Config with pytest.config.Config (#167)

0.2.19

  • matching: full support for reprcompare (#165)

  • add makefile tarets to run databases tests separetly and add split requirements (#162)

  • Matching assertrepr (#163)

  • New experimental assertrepr_compare plugin (#143)

  • Depreacte event_loop and loop fixtures (#145)

  • add testsuite.__version__ (#161)

  • asyncio-legacy: do not close event loop (#160)

  • ci: update outdated ubuntu runner (#158)

0.2.18

  • Remove loop fixture usage (#157)

  • Add badges to README.rst (#154)

  • Recursive partial dict (#150)

  • deps: update packages versions (#153)

  • acallqueue: add func property (#151)

0.2.17

  • Add changelog

  • matching: matching now is available as testsuite.matching

  • matching: specify capture matching rule inside pattern

0.2.16

  • pytest-asyncio 0.25 support while keeping compatibility with 0.21.x

0.2.15

  • mongo: add mongo uri header

  • kafka: implement message headers

  • introduce traceback.hide

0.2.14

  • matching: add any_list, ListOf, any_dict, DictOf and Capture

0.2.13

  • kafka: wait until broker is stopped

  • Update AUTHORS file

0.2.21

  • redis: wait for replicas and master negotiation in sentinel configuration

  • redis: use CLUSTER NODES and CLUSTER SLOTS information to wait for cluster startup

  • hide redundant tracebacks