If you found your way to this article, your company probably sucks in digitization. Discover the 5 signs of sucking in digitization and what you should do about them.
1. Your Managers think they have a plan
The fuck they have. In the fast paced digital revolution basically nothing is under control. Especially not if you’re in a company with more than 1k people. As soon as something is planned and under control, it is probably outdated like a 1995 floppy disk software.
Why do you think every new (cool) software project is being developed in an agile way? Why do you think modern software architecture is more focusing on self-service and adaptability? Because people who actually know what they’re doing understood that you can’t plan everything anymore. As soon as you have something done, there probably already is a new tool/software that will help you improve even further.
And you should embrace that!
There are actually a few smart ways how to best maneuvre through this.
- Use open source as much as possible (this will help you being flexible when switching tools later)
- Think modular (this will help you replace the shit your stupid intern did)
- Think structured (e.g. ETL / ELT) (this will keep processes readible once your ass got fired)
- Think in self-service architectures (empower digitzation and change across ALL departments)
If you really mean it with digitization in your company you should forget about a central team just designing and managing everything in terms of analytics. No matter what they will produce, it will suck for those who need it. Digitization is more of a mindset, a company wide shift, an emergence of digital skills everywhere across the company and a mindest that allows for all that change to happen und chances to be utilized.
The only guiding principles should be keep on learning the new/hot stuff, stay agile, forgive a lot of failed attempts.
2. Your Managers are afraid of "new" tools
Ever felt the need to explain WHY you would use a certain tool? And that you saw it on YouTube isn’t a suitable answer?
"Yeah I'm not a big fan of Python you know, it's too complicated for most of your team members"
Probably he’s not a big fan of his own company as well. Polaroid wasn’t a big fan of digital photography and straight went bankrupt when digital photography became affordable for everyone.
Once other companies employees are capable of using all sorts of programming languages to finally speed up their shitty processes yours is fucked. Well deserved.
I’m not saying that every of your problems can be solved with some sort of new tool but whenever it can be, there’s a high chance it will do it in a highly economic way. Especially if we’re talking about open source solutions for your proprietary problems.
Especially if we’re talking about one of your own employees approaching you proactively, willing to solve your shit, willing to learn and stay engaged.
Let him work, give him a budget for a nice little DevOps online course, give him some time, let him fail and encourage everybody else to follow the journey. Ultimately you will be collecting the reward.
If a Python project is set up in the correct way, it will solve your tasks automatically, transparently, repeatedly and in a documented way. Solutions will be versioned and documented for you. Even if your man gets run over by a bus, the code will stay, keep solving your shit and the next employee (if he’s not completely dumb) will be able to figure out what is being done there.
Of course, if nobody in your company never gave a shit about programming, then somebody downloading things like “Python” and “Anaconda” or “Spyder” and asking for a Proxy User to download “Libraries” might sound scary to you. But isn’t that the same type of scary if somebody in the 80’s said you should no longer print your letters and trust all your communication is safe somewhere in the “cloud”?
3. It takes endless discussions to get data access
If you’re building something or trying to find something out by digging through data and suddenly run into a wall there’s a high chance this has either to do with you not knowing how to access the right data or you’re needing the correct reading rights.
Both should not be the case in a modern corporation, yet are the standard practice basically everywhere and also the reason why so many projects and attempted insights into the data fail.
Since 2020, data engineering books are full on how data warehouses are outdated and the data ecosystem and architectures need to move on to better structures. The reality is, in 2020 and beyond I bet more than 2/3 of all companies are not even operating proper data warehouses that can be used by analysts.
They are having a weird collection of operational systems and a random mix of databases that store some of the operational data for analytics purposes but only accessible by a very low number of people.
Companies are mixing up basically everything here to a random Spaghetti-Discussion ball of shit.
- Operational vs. Analytical systems
- Data access / data protection fears vs. a good permission system
- Data producing units, data consuming units vs data producing and consuming units
I’m just spitting out all the important buzzwords here. But in your data access discussions you will most probably encounter most of them in one or the other form.
“We’re not giving out the read-access to this database as it will be bad for its performance“
– Hey! I want to do analytics, from an analytics database. I do not want to access any sort of operationally relevant system. Either you’re doing all of your own analytics on the operatinal database (which you shouldn’t be) or you implemented a shitty database for analytics in the first place and never thought somebody would actually want to use it.
“We’re worried the data might fall into the wrong hands, thats why you cannot have read access“
– Lol, I am working in the same company as you and I have a clear usecase for the data. Most probably it will make us a fortune and we will all get a raise. Is this what you consider wrong hands? Probably the issue is rather that you’re just sitting on a dumpster of chaotic data with no idea of its confidentiality and no clear user model of a clean read-access.
“Why can’t you just download it and work it out in Excel?“
– I actually want to MAKE something out of the data. I don’t want to just look at it every day before bedtime or read it to my children. If have a USE CASE that I want to automate, monetize and push forward. I need current data, I need it 12 times every day and I want to push my results further into my result database. No, Excel won’t do. I am consuming data but in a faster pace than all of the other idiots here combined and then I am PRODUCING information that will actually generate value.
4. You know at least one guy who's writing cool VBA macros
“Hey, I know how to automate that with VBA”.
“Hey, I’m taking this VBA course that will help my team get stuff done faster”.
If you hear those sentences call the police.
If you ask me, VBA is the incarnation of a fucked up digital working environment in the 2020’s. I’ve heard of companies where people use VBA just because all other modern programming environments are blocked on their computers and they’re too frustrated to call the IT people.
Don’t get me wrong, VBA probably saved the world multiple times from a collapse of the financial system. But slowly it’s time is up. The reason for that is simply that there are so much better an stable ways to hack nowadays, even for non-Dev-guys like me.
Also yes, earlier I said, one should be open for any sort of tools. But I also said you need to keep on learning the new stuff.
Think of a normal CI/CD flow and a small vServer. You could implement source control, automated testing, code documentation, builds and deployment even with your teams not even noticing but just getting feedback that everything runs smoothly.
With VBA in most cases you’re just relying on some random people opening their laptops, finding the right version of the file, copy pasting the right data into it and hoping nothing breaks if you hit the execute button.
5. You're getting weekly report files that you need to process
Probably you have one or more E-Mails per week containing a file that you need to generate some sort of report or analysis. Ever questioned where this file is coming from? What sort of system or data is behind that file?
The issue with files and reports that are created like this are that they are in many cases a blackbox to most of the recipients which causes a lack of trust for their contents. Secondly there is most probably some sort of unknown stakeholder into the process that has either something to do with the file creation directly or with the process of distributing the file.
Changing the process will need this unknown stakeholder to be reachable in the first place, agree to changes, do work, … and then you still have a more or less black box file. Or imagine that unknown stakeholder already left the company and nobody actually knows what the underlying process is. Maybe there is just some poor dude in your India office repeating the steps he was told to do until the end of time. This can fuck up any change process severely as nobody dares to turn off the process but nobody actually can really change it.
Second big challenge is – say you really need to do some analysis based on the files that are sent to you. You still would be sitting on a bunch of weekly files. Would you hire an intern to combine them all? What if the layout changed in between? What if you made a great analysis about last Q4 and then your super smart manager likes it and asks you to do a Q4 analysis for the past 10 years? (Bet you wouldn’t even be able to find data of the past 10 years).
Here’s a better world: A self-service data-something (warehouse/lake/mart) that allows the users to directly connect to a datasource they need with a programming language of their choice. An intelligently designed dev-workflow that forces the users to document and test their code before rolling it out. A company wide repository system that keeps track of all changes and sufficient meta-data to actually judge on the accuracy of your report.