AI - 021
Mission
Build an AI system whilst shipping value-adding intermediaries.
The above (stolen from the company trying to solve self-driving cars: comma.ai) statement, or something like it, is what I believe companies getting started should be aiming for.
This approach allows a team to demonstrate the value of AI, whilst learning and iterating as they go along, building something for the future.
Why not just ship models?
Although this will add value quicker, it’s not a good idea in the medium- and, especially, long-term. You will run into all sorts of problems. Like the build-up of ML-tech-debt, monitoring issues, audit difficulties, data/feature storage redundancy, lack of performance tracking, etc.
Why not just build a system?
This is actually a good idea if you have loads of expertise, resources, and time. Unfortunately, this is not typically the case when you’re trying to get AI started at an organisation. Without all these three components, a top-down approach will likely fail. You’ll find a superior approach half-way through building, higher-ups will bin the project because you haven’t delivered anything in 6 months, economic conditions will necessitate re-focus to existential projects, etc.
Advantage
What does the immediate future of the AI space look like?
This is, obviously, impossible to predict. But what we can do is make some broad extrapolations and think about how these impact what we’re doing.
Prediction: large AI projects will continue to improve in terms of performance, cost, breadth, availability, and modalities.
Does that mean it’s over for us? Why build anything when we can simply purchase an OpenAI subscription? Well, because these companies, and by extension their modals, have neither our proprietary data, nor our specific knowledge. They don’t know our customers as well as we do because they don’t have their data, and they don’t have the same accumulated experience of dealing with those exact customers.
Hypothesis: models that incorporate proprietary data and specific knowledge will be better than off-the-shelf solutions.
I have deliberately used a broad word here in “better”. I think that off-the-shelf models could be more performant, or, more generally, “better” in some specific way. But proprietary models will be “better” in their totality, taking into account cost, ease of deployment/monitoring, compliance considerations, etc. Note also that 1) we may use OTS solutions in the short-term and 2) we can use OTS models as base models for fine-tuning or RAG or something else.
Principles
Prologue 1: principles should have a cost to them. Something like “hard work” is a pointless pile of crap because there is no real downside to working hard. No one would disagree with that principle. But something like “keep it simple”, although these days eye-rollingly common, is much better because there are negatives to keeping things simple.
Prologue 2: principles should be specific to the company and team to reflect your values and culture.
Outcome-led
Bit of a cliché, but a cliché for a reason. In data science it is very easy to get excited by - and wrapped up in - some new, fancy technique or paper or way of doing something. Failure to remain focused on the value proposition can be fatal.
Iterative
Another way of putting this is experimental. Everything we do, from building the ML system to data gathering, is done in this manner. This allows us to both get (the basic version of) things done quickly (ensuring we deliver ‘good enough’) and produce (ultimately) high-quality output. It also benefits a relatively inexperienced team: we can try a bunch of different things and build on what works.
Robust
In ML there is an obsession with optimisation. We can make this process quicker. This algorithm improves performance by 2%. This way of storing data slightly reduces storage requirements.
These are fine, and necessary, at the cutting edge. Researchers should be worrying about squeezing everything out of a model. Meta’s AI Research Team should be concerned with marginal improvements.
But when you’re trying to get AI started, you should be more worried about building robust systems. Do we have sufficient model monitoring, testing, redundancy in data pipelines, etc. to ensure both that something is not likely to break and that we will be alerted if it does. You will likely be met with a weird mix of excitement and scepticism when first introducing AI so making sure your models work as intended is vital.
Transparent
One other way to combat that is by being as transparent as possible. What projects are you working on? How are they going? What are some resources to learn about this stuff?
Documentation is key here.
Transition
When you start, everything will be a bit messy and disorganised and small. It’s useful to think about what the end-state of these components will be so you can aim in that direction.
- Small and simple –> large and complex. Start simple, eat the low-hanging fruit.
- Suggest –> decision. Models shouldn’t be taking actions until they are of sufficient quality and the surrounding infrastructure is sufficiently mature.
- Internal –> external. Start with internal use cases. If something goes wrong, it’s only Jerome from treasury who is annoyed - you don’t lose paying customers.
- Specific and narrow –> end-to-end. Start with specific problems that require specific models. But end-to-end solutions are better and, once the requisite infrastructure is in place, should be prioritised.
- Classic ML use cases –> everything’s a learning problem. Again, eat the low-hanging fruit, the stuff that SMEs expect and are comfortable with. But then look at other areas. And eventually every area.
- Role overlap –> specific roles. At the beginning, everyone is likely to be a sort of end-to-end data scientist. But as you progress roles and responsibilities will naturally separate.
- DS team –> everyone. When you first start, only your (probably small) team will have any involvement in anything ML. However, I believe everyone should have the ability to participate in ML at an organisation if they want to. Everyone should have the ability to explore a data set and create a basic model. This might be a little idealistic, but I think it’s what you should be aiming for.
Culture
As important as the engineering, as important as the value generated and explained to stakeholders, is the construction of a culture that is conducive to AI.
How do you build one?
Expanding concentric circles of involvement
As I said in the previous section, I believe everyone should have some level of competency with data science. But it’s a bad idea to try and roll this out to everyone all at the same time. It’s better to slowly do this by expanding from core expertise:
DS Team –> Data –> Other Analytical Functions –> Organisation
Each of these requires a different approach. You will need buy-in from all to get AI to work.
The DS Team require projects to work on, constant training and education, and to remain focused on the in-flight projects. The data team need to be up-skilled. Analytical functions need specific education (for example, finance people may need up-skilling, engineers may need education (like don’t just expose an OpenAI end-point to customers)). Product managers need to be marketed to. The organisation just needs some high-level awareness.
Marketing and education
This cultural effort requires a huuuge amount of both education and marketing of the team and the benefits of AI.
A little bit of knowledge is a dangerous thing and everyone in the organisation will probably have either a very narrow or just plain wrong view of AI and how to use it. It’s up to you to correct this by leveraging educational meetings/talks, documenting as much as possible, writing internal blog posts, etc.
But at the same time you must evangelise AI. It’s actually an easy sell: we can increase profit, make work more interesting, and allow us to elastically scale by using this - who’s in? But this has to be explained and re-iterated constantly.
Case studies (doing the work)
The easiest way to do this is to point to existing projects that have validated these claims.
Having these 1-3 case studies (or POCs, but these work less-well) sort of does the selling for you.
Tangible results, and the infrastructure and documentation that come with them, are vital in terms of selling to the rest of the company.
ExCo buy-in and championing
All of this, everything that I’ve talked about, is pointless…if ExCo don’t like the idea.
ExCo buy-in is a necessary condition for moving from 0 to 1.
The best way to get this buy-in is by having a person on the inside. The message is a lot more palatable coming from a peer.
And, contrary to what I said above, it’s a difficult message to deliver in a lot of ways. The development of an AI system requires large amount of time and resources, and the main benefits of such a system are only realised in the long term.