Everybody makes mistakes. But if you make too many mistakes, there's definitely something to learn from those mistakes. We had a hell of a last week, and because we were reminded of some pretty important IT-rules again, I wanted to share these lessons with you.
#1 - Upgrade Joomla! 3.2.1 to 3.2.2 without testing
So let's get in what the most stupid mistake first. Everybody tells you that you should always test things thoroughly before doing a major upgrade. That's what we tell our clients as well. But ofcourse, as guru, we feel the rules don't apply to ourselves. So instead of setting up a test site and testing a Joomla! upgrade there first, we simply hit the upgrade button to upgrade our Yireo site.
There were some PHP warnings that popped up, and some small custom made extensions that appeared to be incompatible with 3.2.2. But this was all expected. Unfortunately the 3.2.2 package did not contain all the needed files to complete the upgrade on our site. This gave numerous PHP Fatal Errors, and caused immediate downtime of this site. Things were fixed quickly by uploading the 3.2.2 full package files. But still things got pretty stressful.
#2 - Implement custom routing caching without testing
This is a cool one. Our Yireo site is using a pretty extensive menu with over 500 Menu-Items and a lot of these Menu-Items are pointing to custom components. Additionally we are making only little use of the Menu modules of Joomla! but implement custom code to create dynamic menu navigations. But all this was performing quite well but not optimal. To get the best out of this, we implemented caching of Menu-Items but also caching of routing paths for the custom components.
After implementing this change on the live site, we noticed 404-errors began to gather in the logs. We quickly fixed this (and now enjoy a faster site) but still the site was not behaving as it should for about 1 hour. Again a calculated risk, but we could have easily implemented this in a testing site, and simply push all changes using Git, which is fully operational on our site, which again makes this messing with the site even more stupiduous.
#3 - Upgrade PHP-FPM without testing
We were not done yet with messing around. After the downtime with the upgrade to Joomla! 3.2.2 we decided to shoot ourselves in the feet somewhere by upgrading PHP-FPM with a simple yum command. Unfortunately this also ment that the main PHP-FPM configuration was outdated because of a changed syntax, which - you are guessing it already - was causing downtime.
#4 - Configured Magento cookie domain www.example.com/
The final mistake was actually the least troublesome and also on a Magento demo-site that does not get much visitors. We configured the cookie domain with a backslash on the end, no longer allowing us to login to either frontend or backend.
Without testing
I'm not sure if you catched the last 2 words of all the 4 headings above: Without testing. Our week would have been a lot less stressfull if we had first applied the first rule of IT-club: Test things first. Perhaps it would have taken us a bit longer to implement things - well, it would definitely have taken us longer if we would have tested everything. But also a lot less stressed, and less stress makes us live longer.
We are now putting post-its everywhere with a simple note: Test first. We had a big laugh about it though, because we can still do whatever we want.
About the author
Jisse Reitsma is the founder of Yireo, extension developer, developer trainer and 3x Magento Master. His passion is for technology and open source. And he loves talking as well.
