[TIP] pytest-sugar -> pytest

[Stefan Scherfke]

Hi all,

I recently stumbled across pytest-sugar
(https://pypi.org/project/pytest-sugar/).

This plug-in produces quite similar, but slightly nicer/more clear
output. Some examples:

- different highlighting for test directories / files / functions, which
makes it easier to visually group the output.

- Green checkmakrs (✓) instead of dots for passed tests.

- When run with "--capute:no -vv", pytest produces output like
"{testname} {captured output} {outcome}" while pytest sugar prints
"{caputured output}\n{testname} {outcome}" which is IMHO more
readable.

This is all nothing big or special, but in sum it improves the
readability of pytest’s output quite a bit.

I think it might make sense to merge/integrate pytest-sugar into pytest
(or just modify pytest’s default output format). What do you think?

Cheers,
Stefan

[Florian Bruhin]

Post by Stefan Scherfke
I think it might make sense to merge/integrate pytest-sugar into pytest
(or just modify pytest’s default output format). What do you think?

-1 from my side. I've seen it fail in funny ways in some special
terminals (even unicode output can be problematic). I'd rather have a
default output which actually works in various scenarios.

I think smaller incremental changes (like the progress output which was
added) are nice, but pytest-sugar as a whole comes with too many
gotchas.

Florian

[Stefan Scherfke]

Post by Florian Bruhin

Post by Stefan Scherfke
I think it might make sense to merge/integrate pytest-sugar into pytest
(or just modify pytest’s default output format). What do you think?

-1 from my side. I've seen it fail in funny ways in some special
terminals (even unicode output can be problematic). I'd rather have a
default output which actually works in various scenarios.

Okay, I can understand that.

Post by Florian Bruhin
I think smaller incremental changes (like the progress output which was
added) are nice, but pytest-sugar as a whole comes with too many
gotchas.

Then about about these incremental changes?

1. It should not be to hard to change the order in which a test’s name,
the test’s output and the test result are printed?

The current default is not too readable:

tests/test_stuff.py::test_xy has this test PASSED OR FAILED PASSED

In the example above, its quite hard to see that "has this test
PASSED OR FAILED" is the tests output and not pytest-output.

We could copy pytest-sugar's formatting to make it more readable:

has this test PASSED OR FAILED
tests/test_suff.py::test_xy PASSED

2. Since pytest already can color its output, we could color
the test dir and test file (in non-verbose mode) or test-path and
test-func (in verbose mode) in different collors.

3. We could also show the progress in verbose mode.

4. We could colorize the dots and Fs of test results in non-verbose
mode (e.g., green ".", red "F", yellow "x", …)

5. Another idea would be to allow the user to provide templates for
test output (similar to what you can to with hg/git log formatting)

These would be relatively small changes which don’t add any new
dependency or a lot of complexity to pytest but would improve the UX.


Cheers,
Stefan

Post by Florian Bruhin
Florian
--
GPG: 916E B0C8 FD55 A072 | https://the-compiler.org/pubkey.asc
I love long mails! | https://email.is-not-s.ms/

[Bruno Oliveira]

Hi Stefan,

Post by Stefan Scherfke
Then about about these incremental changes?
1. It should not be to hard to change the order in which a test’s name,
the test’s output and the test result are printed?
tests/test_stuff.py::test_xy has this test PASSED OR FAILED PASSED
In the example above, its quite hard to see that "has this test
PASSED OR FAILED" is the tests output and not pytest-output.
has this test PASSED OR FAILED
tests/test_suff.py::test_xy PASSED

What do you mean by "the test’s output"? You probably don't mean things
like print() and such as those are captured and shown at the end of the run.

Post by Stefan Scherfke
2. Since pytest already can color its output, we could color
the test dir and test file (in non-verbose mode) or test-path and
test-func (in verbose mode) in different collors.

Sounds good!

Post by Stefan Scherfke
3. We could also show the progress in verbose mode.

We currently show the progress (percentage or count) in verbose mode. Do
you mean a progress bar?

Post by Stefan Scherfke
4. We could colorize the dots and Fs of test results in non-verbose
mode (e.g., green ".", red "F", yellow "x", 
)

Sounds good as well.

Post by Stefan Scherfke
5. Another idea would be to allow the user to provide templates for
test output (similar to what you can to with hg/git log formatting)

This personally feels like a little too bloat, but if others think this is
nice then OK. This would be an option specific to the terminal plugin.

Post by Stefan Scherfke
These would be relatively small changes which don’t add any new
dependency or a lot of complexity to pytest but would improve the UX.

Those are all plausible ideas, feel free to create issues to them in the
tracker. Furthermore, some of them, specially 4, are really simple and
candidates for a PR. :)

Cheers,
Bruno.

[Stefan Scherfke]

Post by Daniel Knüttel
Hi Stefan,
Then about about these incremental changes?
1. It should not be to hard to change the order in which a test’s name,
the test’s output and the test result are printed?
tests/test_stuff.py::test_xy has this test PASSED OR FAILED PASSED
In the example above, its quite hard to see that "has this test
PASSED OR FAILED" is the tests output and not pytest-output.
has this test PASSED OR FAILED
tests/test_suff.py::test_xy PASSED
What do you mean by "the test’s output"? You probably don't mean things like print() and such as those are captured and shown at the end of the run.

That’s exactly what I mean. :-)

This is only relevant, if you invoke pytest with "--capture=no --vv",
though (e.g., in a CI pipeline).

Post by Daniel Knüttel
3. We could also show the progress in verbose mode.
We currently show the progress (percentage or count) in verbose mode. Do you mean a progress bar?

No, the progress in percentage would be okay. When I last tried it,
I only got progress with non-verbose mode but not with "-vv". Dunno,
have to check it again. If you’ve got the percentage, a progress bar
doesn’t add to much additional benefit, IMHO.

Post by Daniel Knüttel
Those are all plausible ideas, feel free to create issues to them in the tracker. Furthermore, some of them, specially 4, are really simple and candidates for a PR. :)

I will see what I can do. :-)

Cheers,
Stefan

[Florian Bruhin]

Post by Stefan Scherfke

Post by Florian Bruhin

Post by Stefan Scherfke
I think it might make sense to merge/integrate pytest-sugar into pytest
(or just modify pytest’s default output format). What do you think?

-1 from my side. I've seen it fail in funny ways in some special
terminals (even unicode output can be problematic). I'd rather have a
default output which actually works in various scenarios.

Okay, I can understand that.

Post by Florian Bruhin
I think smaller incremental changes (like the progress output which was
added) are nice, but pytest-sugar as a whole comes with too many
gotchas.

Then about about these incremental changes?
1. It should not be to hard to change the order in which a test’s name,
the test’s output and the test result are printed?
tests/test_stuff.py::test_xy has this test PASSED OR FAILED PASSED
In the example above, its quite hard to see that "has this test
PASSED OR FAILED" is the tests output and not pytest-output.
has this test PASSED OR FAILED
tests/test_suff.py::test_xy PASSED

I wonder how feasible that is (with the test output) without having to
capture output. Not sure why this difference even occurs - does
pytest-sugar only print the test name after the test has finished?
If so, that might introduce a new UI issue: You don't clearly see which
test is running right now.

Post by Stefan Scherfke
2. Since pytest already can color its output, we could color
the test dir and test file (in non-verbose mode) or test-path and
test-func (in verbose mode) in different collors.

That seems like a nice idea!

Note that some colors won't work well depending on the background color,
but what about making it bold?

Post by Stefan Scherfke
3. We could also show the progress in verbose mode.

That already is the case here, with pytest 3.6.4.

(I'd like to upgrade to 3.8, but
https://github.com/pytest-dev/pytest-bdd/issues/250 prevents me from
doing so - anyways, different story :D)

Post by Stefan Scherfke
4. We could colorize the dots and Fs of test results in non-verbose
mode (e.g., green ".", red "F", yellow "x", 
)

I'd probably keep the dots in the default color, especially if the test
names and all that is already colored. Pointing out the (hopefully) less often
printed chars seems nice though.

Post by Stefan Scherfke
5. Another idea would be to allow the user to provide templates for
test output (similar to what you can to with hg/git log formatting)

Not sure how much you can cram into a template string. At some point
it's just easier to write code and write a plugin.

My $0.02 ;-)

Florian

[Daniel Knüttel]

Hi Stefan,

I think sugar should be considered a customization that should not be
integrated as the default layout. There are several reasons:

1. The use of funny characters. Pytest is one of the most basic tools
for developing anything in python. It should work properly on any
platform and I think that the funny characters would break that.
2. I think the default pytest output is easier to read, especially when
it is integrated in another command (like `make test`). And the error
messages are easier to read (in my opinion).


However I think that there are some changes that could be adapted to
the default pytest output:

1. Use colors for highlighting the file name. That would be great.
2. The list of failed/error'ed tests in the end is great.

The option `--capute:no` did not work for me.