TESTING_AND_QA.md may require a DEPLOYMENTS.md
Copied verbatim from an email I sent
My major questions are around some definitions, and how they apply to this. I'll go over them in a solid train of consciousness manner :)
What is a product? In the simple case of ARDC, there's three things that OTS has a hand in delivering:
- The hypha codebase
- The packages/operating system that make up the running machine
- The living configuration of the hypha system
One of these is very straightforward when applied to the document as written (the first). But when applying some of the practices to the other two, one starts to run into challenges. How do you tag the "release" of the packages update? How do you "deploy" to staging the living configuration of the hypha system? I understand that you can just go into staging and make the same changes that you would be making to production, but deployment feels like something that is technically reproducible and taggable (ie, checked in to a codebase somewhere), at least in keeping with the spirit of the document.
When you apply this kind of a question to torque, it gets quite complicated quickly. I can write a whole email about that, and how we would have to change what torque even means to effect this policy on that overall product.
What does deployment really mean? From a technical point of view. And should all things that OTS provides fall under that category? I think there's a temptation to say "a deployment is whatever the client needs that we can deliver" but that feels like "make it up as you go along" to such an extent that the whole process comes undone. If OTS has a standard of what it means to deploy something, and that definition is clearly communicated to the clients, and it lends itself to the QA/Testing process, then lives will get easier down the line.
What does staging mean? We talked about this some in our followup meeting, and settled on an answer for ARDC. That it's a duplication of production with all data that could be sensitive replaced with dummy information. How is staging different than demo? Should they be separate always? What is the process around maintaining staging? Should there be a clearly defined process?
In my ideal world, I would go so far as to say that the systems are all defined such that no developer, or even a person, is allowed to go on a machine and just "make it happen." We use ansible for every product, with everything checked into a repository somewhere. If the change you need to make isn't possible through ansible, then you don't get to make it (in reality, all things are possible through ansible because you can just put a script over and run it). Ansible scripts declares the software that's on machines, and does things like the database sanitization.
I'm coming around to thinking that this document can't stand alone, but is built on a theoretical DEPLOYMENTS.md that clearly defines all of these things.