[TIP] Feedback request: flake8-aaa

[James Cooke]

Hi all,

If you're using something like the Arrange Act Assert pattern (AAA) /
Given When Then for writing your Python tests and Flake8 for linting
your code, then I'd be interested in your feedback on a Flake8 plugin
that I've been working on called flake8-aaa:
https://github.com/jamescooke/flake8-aaa
This plugin is in beta and currently checks four basic rules to help
break a test into three distinct blocks:
https://flake8-aaa.readthedocs.io/en/stable/rules.html and is based on
the description that I've put together for AAA here:
https://jamescooke.info/arrange-act-assert-pattern-for-python-developers.html
.
Some background: I wrote the plugin because I found that I was
slipping when following the AAA pattern and Adam (who some of you
might know https://github.com/adamchainz/) suggested that I write a
Flake8 plugin. It's currently kind of self-hosted (in that it tests
its own tests at test time), plus I'm using it happily at work and in
some open source projects.
Please let me know via email or GitHub issues if:
- You like the idea and want more!
- You have any tests that failed any of the current four checks, but
think they shouldn't have.- Conversely, you've found a test that passed, but you think it
shouldn't have.- This kind of thing is interesting, but you'd prefer a plugin
for pylint.- You've spotted that my plugin's code is terrible and how I should
improve it (PRs welcome ;) )- There's already something that does this / some other package that I
should be putting my efforts into.
Thanks for reading and happy testing!

James

[Ben Finney]

[…] I'd be interested in your feedback on a Flake8 plugin

Thanks for writing a style checker for test code.

This plugin is in beta and currently checks four basic rules to help
https://flake8-aaa.readthedocs.io/en/stable/rules.html

That page doesn't make clear to me: What consititudes a valid block of
each kind?

For example:

* Must the “Act block” be an assignment? Are other statements allowed?
Are multiple statements allowed? What exactly will be *not* allowed as
an Act block?

* What exactly constitutes a valid Assert block? What styles of
assertion are recognised (pytest, nose, standard library `unittest`,
others)? What constitutes an invalid Assert block?

Please let me know via email or GitHub issues

I don't want to use GitHub, so I hope you'll be participating in this
forum where you asked for feedback :-)

[James Cooke]

Hi Ben (and list),

Post by Ben Finney
Thanks for writing a style checker for test code.

Thanks for the positive feedback :D

Post by Ben Finney

Post by James Cooke
https://flake8-aaa.readthedocs.io/en/stable/rules.html

That page doesn't make clear to me: What consititudes a valid block of
each kind?
* Must the “Act block” be an assignment? Are other statements allowed?
Are multiple statements allowed? What exactly will be *not* allowed as
an Act block?

Thanks for checking out the rules page on RTD. My intention with this page is that it should serve only a record of the checks currently carried about by flake8-aaa with some help about how to failures can be fixed - it is not intended to be a description of the AAA pattern. Instead, the list should be read in the context of the outline on the AAA pattern for Python developers, just as pycodestyle's error codes list (https://pycodestyle.readthedocs.io/en/latest/intro.html#error-codes) should be read in the context of PEP8, etc.

So to start to answer your questions on Act, please see my original outline of the AAA pattern: https://jamescooke.info/arrange-act-assert-pattern-for-python-developers.html#act

Post by Ben Finney
Must the “Act block” be an assignment? Are other statements allowed?

As of version 0.4.0 of flake8-aaa, there are three options:
* assignment `result =`
* pytest exception raised `with pytest.raises(...)`
* unittest exception raised `self.assertRaises(...)`.

Post by Ben Finney
Are multiple statements allowed?

I'm interested to see an example of a test that requires multiple statements as the action. If you could share one that would be helpful. In my experience where I've come across actions with multiple statements, either the first statements are really arrangement, or the latter statements are really assertion / result manipulation, or a combination of both.

Post by Ben Finney
What exactly will be *not* allowed as an Act block?

The developer can put anything they want in a Act block if they put ` # act` at the end of the statement. I've created that functionality to serve a bit like `#noqa` and also to allow for work arounds while the plugin is in beta. Is there some type of statement you would like to see banned from Act blocks?

Post by Ben Finney
* What exactly constitutes a valid Assert block? What styles of
assertion are recognised (pytest, nose, standard library `unittest`,
others)? What constitutes an invalid Assert block?

For context, there is the definition of Assert from the AAA pattern: https://jamescooke.info/arrange-act-assert-pattern-for-python-developers.html#assert

Right now all assertions are recognised because I've made the plugin super lazy - it counts **all** the code after the Act block as the Assert block. My plan once the outstanding issues with Act blocks are fixed (https://github.com/jamescooke/flake8-aaa/issues/) is to add a check that there are no unnecessary spaces in Assert blocks, then to check that an Assert block exists for Act blocks that have an assignment.

Post by Ben Finney
I don't want to use GitHub, so I hope you'll be participating in this
forum where you asked for feedback :-)

I'll do my best!

Finally, in case it's helpful, there are some unittest test examples to an `examples/good` folder of the project repo. The plugin is tested against these at test time to assert that it does not complain about anything there: https://github.com/jamescooke/flake8-aaa/tree/master/examples/good

Hope you find those answers helpful - let me know your thoughts.

Cheers,

James

[Daniel Knüttel]

Hi James,

Your plugin seems to be really helpful, especially for young
programmers. I am pretty sure that I will enforce the usage of this
plugin for my junior developers :-P.

Just like you I think that formalization is a big improvement for all
parts of software development but most tests are more like a dirty hack
to get stuff done although it is most important to understand them fast
and perfectly.

By the way: do you know a way to help young developers to understand
test driven development and to implement it in their workflow?
I often face the problem that they use a python shell or a small "main
script" to debug their code instead of writing actual tests. Which is
extremely annoying in a team, in particular when they ask you for help
and there is no test (or the "main script") on the VCS.

Daniel

Post by James Cooke
Hi all,
If you're using something like the Arrange Act Assert pattern (AAA) /
Given When Then for writing your Python tests and Flake8 for linting
your code, then I'd be interested in your feedback on a Flake8 plugin
that I've been working on called flake8-aaa: https://github.com/james
cooke/flake8-aaa
This plugin is in beta and currently checks four basic rules to help
break a test into three distinct blocks: https://flake8-aaa.readthedo
cs.io/en/stable/rules.html and is based on the description that I've
put together for AAA here: https://jamescooke.info/arrange-act-
assert-pattern-for-python-developers.html .
Some background: I wrote the plugin because I found that I was
slipping when following the AAA pattern and Adam (who some of you
might know https://github.com/adamchainz/) suggested that I write a
Flake8 plugin. It's currently kind of self-hosted (in that it tests
its own tests at test time), plus I'm using it happily at work and in
some open source projects.
- You like the idea and want more!
- You have any tests that failed any of the current four checks, but
think they shouldn't have.
- Conversely, you've found a test that passed, but you think it
shouldn't have.
- This kind of thing is interesting, but you'd prefer a plugin for
pylint.
- You've spotted that my plugin's code is terrible and how I should
improve it (PRs welcome ;) )
- There's already something that does this / some other package that
I should be putting my efforts into.
Thanks for reading and happy testing!
James
_______________________________________________
testing-in-python mailing list
http://lists.idyll.org/listinfo/testing-in-python

[James Cooke]

Hi Daniel,

Your plugin seems to be really helpful, especially for young programmers.

Thanks for the kind words :D

I am pretty sure that I will enforce the usage of this plugin for my junior
developers :-P.

That sounds great - I'm interested to know how it goes. One caveat would be
that, as with all checkers, they can be exploited. Just because a test might
pass the checks in flake8-aaa doesn't make it a "good" test, any more than code
that passes flake8 is "good" code.

By the way: do you know a way to help young developers to understand test
driven development and to implement it in their workflow?

I don't know for sure, but I think that the "Test Driven Development: By
Example" book by Kent Beck is a great resource for this because:

* It introduces the red, green, refactor process. I think that this process is
what you want to replace the python shell and or "main script" usage.

* It shows how DRY can be applied between tests and SUTs. I think this will
help improve the quality of written tests.

* Less related to TDD, it shows the value of writing detailed to-do lists and
finishing a day's work at the red state. These things have helped me be more
efficient when coding and more able to put down and pick up work, especially
helpful when interruptions are common.

My guess is you might also find value with "gatekeepers" - tests that are run
on CVS branches to assert that new code has test coverage and passes code
quality requirements.

Hope that's helpful.

Cheers,

James

[Daniel Knüttel]

Hi James,

I will definitely take a closer look at what you sent, it seems to be
really helpful. I will give you some feedback in some months when I
have some observations. I hope I don't forget it.

Cheers,