When we think about the skills required by developers, we never consider the soft skill to present an idea to others. This is not something easy to train, but it is definitely something that can be trained.
There is not a “correct” way to organize your servers and git repos and each company has its own problem and solutions. Some companies use test servers to run some very basic verifications in their codebase before deploying new features to production. Some even have a staging server that is as similar to production as possible, where new features are tested for a last time before deploying to prod. Those servers usually run deployed code that is in the development, staging, and master/main branches in the git repo. This would be what the usual flow of development looks like:
Another good practice many companies adopt is the versioning of the code to keep track of what is deployed on each server. There is a lot about versioning alone that should be discussed in other posts, but let us adopt the format major.minor.build for our example.
Dealing with Bugs in Tests
Now, let us imagine that a bug was discovered in one of your features when it was being tested in the Test Server. We do not want to merge any code to Staging that has a known bug, but then, how do we deal with it if more features are also being deployed to Test?
The first option we have is to just work on a fix for our buggy feature and deploy all other features once it is fixed:
This is nice because each version number on all servers has the same code, it is easy to merge from development –> staging, and from staging –> master. The only bad part: we have to wait for the fix to Feat. 1 to be ready before we can merge. Now, what happens if Feat.2 and Feat.3 also have bugs discovered? Yes, everyone gets blocked.
One alternative to this problem is to revert the change from Feat.1 as soon as you find it.
This works well because now Feat.1 can take as long as needed to be fixed, if any other Feature has a bug discovered we just revert it, and development can continue. Also, all the versions on all servers have the same code (which means, v1.0.3 in staging is the same as v1.0.3 in development).
What is the problem then? Well, when Feat.1 is done, we have to revert the revert. And our history becomes a mess. Imagine what it would look like if Feat.2 and 3 also did reverts?
There is one other option that works with many costs. We cherry-pick the contents of the development branch into staging.
Now this one comes with a bunch of requirements to work. Let me point out a short list of things to be careful when going in this direction:
When developing your feature branch, you must not merge the parent branch regularly (only when you are going to merge). This adds a risk that your feature branch and the parent branch will be very different when you merge, and your code will need a lot of adaptations to be merged. You must not squash the commits when merging to the parent branch. Otherwise, cherry-picking will be almost impossible. The version number in development no longer guarantees it has the same code in staging and prod. Notice that v1.0.2 in the development branch has Feat.2 and the faulty implementation of Feat.1, whereas v1.0.2 in staging only has Feat.2. Code in staging might become very different than code in develop over time. One thing to consider is to merge staging –> development on a regular basis. But it also has the nice advantage that all the features can be deployed from development to staging independently, with little concern about bugs introduced by other features.
Alternatives for Testing Large Features
Finally, for very large features that we expect to find many bugs when testing, we can consider another approach: deploying a branch in a separate server and running tests there. You can configure this server to use all the dependencies ( database, storage, redis cache, etc) of the test server.
This works great and keeps our repo clean because you will only deploy to development once it has already been tested by the test team, with the extra cost of having to configure any other tool or client we use for our test with the new server endpoint.
What are the cons? Besides the extra configuration for the test team, there is an extra cost for the new server. Also, if you do not have a pipeline ready to deploy it, there is the extra work of deploying it correctly. Basically: more work to do for this to work.
Conclusions
There is no “best” way of handling bugs found in tests. As a team, you have to figure out the approach that best suits the moment. If you have time, not very large features being developed, stuff that hardly has bugs or that bugs will be easily fixed, just wait for the fixes and deploy it. If the time to fix the issue is not clear, you have to choose between cherry-picking and reverting.
Notice that a great advantage here would be knowing in advance if the feature implementation is large/complicated/has high story points or not. That is one good use of those estimated story points in sprint planning. 😉
