Working in an Agile environment means that we have many releases throughout the year. Despite our sprints being of the same length, there generally seems to be releases where there is a lot to of functionality, and some where not so many changes are going in. Releases tend to be a team effort here, but the overall management of the process falls to me. But no matter how regularly we deploy, or how big an impact the release will be, it always makes common sense to be diligent when preparing for a release. The last thing I want to be doing on a Saturday is trying to fix an issue that could and should have been caught in regression. So how do we help ourselves?
I usually treat deploying to our PreProduction environment in time for regression as my time to check that the deployment process works well. The regression period is as much about the deployment as making sure that all our features are running as expected. So I deploy to PreProduction in exactly the same way that I deploy to Production.
We have an agreement in our team that database and cube changes are checked in and merged to our Main branch with two days spare, so that this leaves me time to deploy and work through any issues. It makes sense that at this point all database & cube changes have been completed and tested, and if any bugs are found then the updates to the database should be minor. Then on the night before regression I deploy the code.
w/r/t code As all our test and dev builds compile in debug, and our regression/production builds compile in release, even the simplest thing like running the build a few days before Regression can help, as the configuration manager might not have been updated for Release, or strong keys not applied.
Testing the deployment only works for me because we have an environment for regression that mirrors that of Prod: we have separate web, database, SSAS and service boxes, albeit far less of them and far less powerful. When I say far less of them I mean that we don’t have a need for running clustered services or load balanced web boxes like we do in Prod, but the architecture of what component sits where is exactly the same as on Prod. Having a scaled down regression testing environment has helped us catch out issues such as double-hop issues with permissions which we can act on before release.
w/r/t permissions, all our service accounts are exactly the same as the permissions in Prod. Client data cannot leave the production domain, so our PreProd environment is in the same domain. This is especially important because on the old PreProd environment the service accounts were sysadmins on the boxes and on the databases… in fact they were probably the most powerful accounts! And on test environments, sometimes when everyone is just trying to get a feature working, some people may let their professionalism slip and will change a permission to higher principles rather than work through the permissions appropriately. This doesn’t help us when coming to release. In fact it is counter productive as we spend time figuring out why the permissions for the service accounts don’t have permissions to access a new schema when it’s been working all sprint, and no one is around to answer that question. Having tight control of a PreProd environment helps us catch those issues.
Our service accounts are also totally different from our production servers. Despite being in the same domain, it makes sense to separate these accounts so that no one can attempt something nefarious via PreProd, or if someone unwittingly locks an account it doesn’t bring Prod down! This might sound obvious to some, but worryingly our old environment share the same accounts across several other environments, on the basis of the fact that as our environments were in the same domain it made sense (to someone) for all the accounts to be shared.
Really, it’s not practical to have every environment mirror that of Prod. It would be very expensive for one thing: Remember that this environment will only be used around 25% of the time, so it’s unreasonable to expect that it’s an environment as powerful as Prod. But if regression is about testing that the potential release does not break what we have in Prod, one way you can really help yourself out here is having an environment that matches production as closely as possible.