2024 Data and AI Trends
Hi Friends,
Happy Holidays! I hope everyone is finding some time to unwind and celebrate with loved ones. I'm excited to share that I'm heading to a warmer climate for my annual Think Week next week. If you're curious about what a Think Week entails, make sure to scroll down for an insight into my approach.
This year has been thrilling, marked by my third consecutive appearance on the Super Data Science podcast, where I indulge in one of my favorite activities - making annual predictions. Additionally, I've contributed to the Data Leadership Collaborative with my yearly forecasts. Predicting the future energizes me, and I'm eager to share this year's insights with you. I'd love to hear your thoughts on them.
Stay curious and enjoy the festive season!
Sadie
Data And AI Trends for 2024
1️⃣ Hardware and Data Hunger
One of the things we know about LLMs in particular – not to mention large machine learning models – is that boosts in performance come from adding the number of parameters we train in a model rather than tweaking that model’s algorithm. This requires more innovative hardware, both chips and servers. I would expect to see more chip players based on the stratospheric rise of NVIDIA as well as a push for more data (and more obscure data sets that we don’t have access to) to grow and flourish in 2024.
2️⃣ The LLM Operating System
One of the reasons for the enormous push for new data and hardware is a platform shift to a new operating system defined by LLMs such as Bard, Claude or GPT. We’ve seen this many times in the past, most recently in the development of the mobile operating system, which generated thousands of apps. Already we’re seeing announcements of app-like models to support niche applications, the prime example being OpenAI’s ChatGPT Store, now slated for release in early 2024. Expect the internet to begin to take on these new model-based characteristics as well.
3️⃣ Thinking Models, Fast and Slow
Just as Nobel Laureate Daniel Kahneman’s best-selling book laid out two ways we think – fast, instinctive, and emotional as well as slower, more deliberative, and more logical – I think we’ll begin to see LLMs characterized by deeper reasoning and slower thinking. Any user of GenAI already has seen the “fast” models, which generate content quickly, automatically, and even unconsciously (hallucinations, unexpected new ideas) but aren’t very good on rational, logical, slower decision-making. I anticipate that in 2024, we’ll start to see the birth of more slower and reasonably thinking models. This trend started to come into focus with a recent paper on training language models with pause tokens, which prompt them to pause and check through some of their answers before presenting them to the user.
4️⃣ Tool Consolidation
Anyone in today’s data space sees how many tools are required – tools for data governance, visualization, cataloging, ETL, and so on. The data trend in 2024 will see the energy continue to shift back toward consolidated master tools, not unlike what Microsoft has been demonstrating with their Microsoft Fabricanalytics platform. I think we’ll continue to see companies release all-in-one solutions. Many will be used in cloud deployments to consolidate multiple functions, and will offer relief to budget-strapped IT and data departments.
5️⃣ Workforce Upheaval
Anyone who has used an AI assistant – what Microsoft calls a Copilot – has witnessed firsthand what a difference they can make. In 2024 we’ll see more and more tools that will be LLM infused or offer some type of AI automation. The anxiety is that this role change is happening at a rate faster than we as humans are able to adapt. People are rightly wondering, What does this mean for my job? How do I adapt? Another legitimate worry is that AI will increase the existing digital divide around the world while at the same time calling the necessity of a four-year college degree into question. Will you even need a four-year degree if you have base skills plus an AI assistant? Although many companies still resist this shift, the evidence from using AI tools to complement skills is that B-level players can more easily upskill to A-level performance.
A call for technology optimism
I continue to be a technology (and AI) optimist, which is why I hope more companies create safe spaces for LLM use and training in 2024, even if it’s not yet part of everyday business. Remember, 40 years ago computers entered our world as workplace “co-pilots.” They haven’t taken over, just made things run faster and more efficiently. That’s why all data leaders should always be looking to the jobs, the goals, the education, the training, and the upskilling that we need to manage what happens in our workplace. Allow your team and colleagues to follow their curiosity about what data can do, and regardless of the data trends we see in 2024 it’s an easy prediction to say you’ll have a brighter future.
It’s Time For A Think Week
In my experience, taking a 'Think Week' has been a game-changer for personal and professional growth. It's a week where I go off-grid, cancel all meetings, and deeply reflect and plan, free from the usual work distractions. I've found that preparing with minimal materials and sometimes changing my environment enhances the experience. The process involves reviewing and reassessing everything from my feelings to business metrics. I break down my reflections into reviewing past performance, resetting strategies, and refocusing on priorities for immediate, short-term, and long-term changes. This practice not only refreshes my perspective but also ensures that I return to my team with actionable, well-considered plans. It's an invaluable tool for any entrepreneur or leader looking to stay ahead and maintain clarity in their vision.
If you want to read more about my methodology for a think week, check out this article.