2025-06-16
(last updated 2025-06-24)
ai
llms
vibe-coding
Last week I had an extremely thought provoking conversation with a careers adviser from 80,000 Hours. He asked me a lot of interesting questions and in particular he pressed me quite hard on why I'm hesitant to get into the AI safety space. I didn't have a great answer in the call and I can tell that I have a lot of complex feelings on the topic that I need a solid way of expressing. This article is written to give myself an opportunity to consolidate and express my feelings.
In this article I will attempt to avoid talking about things that I haven't given a lot of thought, because positions that are not well thought out yet are not worth presenting. I also acknowledge that a lot of the arguments below are based on how I feel right now as opposed to cold hard facts about the state of the world. Again, this is because I am attempting to convey why I am hesitant and having feelings is an occupational hazard of being human.
Starting off with an easy one, I have just moved to Bristol with my partner and most of the AI safety industry is currently situated in London. I'm not saying I'm never going to move back to London, but so far I'm having a pretty good time in Bristol. I think point this is pretty easily countered by organising AI safety stuff in Bristol and semi-regularly attending events in London (this is a direct recommendation from the careers adviser). I think something like this would probably be a very welcome addition to the Bristol Tech Hub community and I'd probably be more than happy to organise some events with them. All this being said, it's definitely some extra friction that makes it harder and I think not having a huge community already will put a lot of work on me to start building that environment.
I am getting the impression that if I wanted to work in the AI safety space, in order to be competitive, I would have to use AI tools. I think there are probably far more reasons I'm AI-averse in my workflows, but the top few reasons are as follows.
I am psychologically allergic to anything that has huge amounts of hype surrounding it. I've fallen into the hype trap with many things before only to be extremely disappointed. I'm an r/patientgamer and I detest 99.99% of the stuff produced by the hype of the web 3.0 movement. I see a lot of similar promises and marketing coming from the AI space and although it seems like a lot of it has more credence than the cryptobros, it makes me uncomfortable how much non-technical people are screaming and shouting from the rooftops about it.
I think this argument has been done to death at this point, but it's worth reprinting in my own words here. I'm early in my career. This means my priority should be learning as much as I can. One of the best lecturers at my university (Dave Wood for any Warwick maths alumni) really hammered home that "maths is not a spectator sport". By this he means that the only way that you really learn anything is by engaging deeply with the problems in front of you and coming up with your own solutions, not just reading the solutions from the textbook. Offloading huge swathes of code-writing to agentic systems is like reading solutions from the textbook. If I write the code myself, I learn far more about my tech stack, the systems I'm working on and, most importantly, the business case for the feature.
I think there is a tradeoff about figuring out how to use AI effectively for learning, but I don't find the LLM feedback loop engaging. This means that I don't wake up with any innate desire to actually try and hone this skill. Of course this is something that can be worked on with time and some persistence, but I just don't have much motivation because I don't find it as satisfying as sitting with a book/docs written by the experts. In short: I really like learning and using AI systems for anything means that I learn less.
I want to start this one by highlighting that I don't think software engineering will stop being a job title, but what I do think is highly possible is that SWEs will start writing less and less code and they will just start becoming a translator for the product managers into actionable technical items which will then get delegated to Claude Code.
I like all aspects of being a software engineer, both the problem solving with code and the squishy parts where you're trying to draw technical requirements from a member of the product team. I think the reason I like doing both of those things is because of the contrast between the two. I like spending the morning solving fun little logic puzzles to make my code work (read: vim macros to bulk edit code, type definitions that leave the code open for extension, ensuring the code is robustly testable etc.), and then spending the afternoon figuring out what the PM meant on the ticket and learning about the business context surrounding my work.
If agentic LLM systems start taking away the first part, all we're left with is the second part, and I really don't think I will find the second part that much fun if it's all that I'm doing. Am I in a privileged position where I find my job fun? Yes. Do I feel like I deserve the right to protect that part of my job? Also yes.
This is probably a longer post for a different time, but it seems likely to me that we are going to have more issues related to misuse of AI by existing social systems and power structures than we are likely to have issues from capabilities.
Based on an 80,000 hours problem profile on the topic, it's likely that my best contribution would be through technical AI safety. With the above in mind, I don't think that I would be contributing in the way that I think is meaningful. In other words: because I am more worried about social issues than capabilities issues I should be working in AI governance, but I don't think I'm well suited to that type of work.
To roughly quote the careers adviser, "there are huge economic incentives to make sure this technology is embedded everywhere". This is obviously true. LLMs are everywhere already and they're only getting more capable and more integrated with our existing systems. Someone should make sure that this is done safely. It would be pretty catastrophic if the Tesco chat bot was willing to tell everyone how to produce neurotoxins; it's important to understand the capabilities and limitations of these systems if they're going to be deployed everywhere.
80,000 hours estimate that as of 2022 there were ~400 people working on AI safety globally. That is a disgustingly small number given how huge some of the impacts are predicted to be. Even if the predicted impacts are off by several orders of magnitude, it seems silly that we're spending ~1,000 times less on safety than we are on capabilities.
A lot of my positions as stated above likely come from the fact that I am a software engineer from outside the AI space who interacts with lots of other software engineers outside the AI space. Naturally, this means that I hear a lot more anti-AI sentiment than pro-AI arguments. 80,000 hours sent across a huge list of introductory AI safety reading and events that look worth reading and attending. I don't think I can honestly say I have a balanced view on modern AI systems, their impacts and where their capabilities will be in 5-10 years without at least attempting to take this seriously and doing some reading.
I think some of the issues listed above are quite high signal, but some of them are definitely lower signal. AI safety being mostly based in London is a low signal reason not to get involved as there are very clear steps to mitigate that issue, but being reluctant to use AI tools and social safety vs capabilities safety are higher signal.
I also think there is likely to be a lot of cognitive dissonance wrapped up in the words that I've written and the points that I've articulated. Working in AI safety would represent a large change in my character from someone who isn't particularly interested in the technology to someone who interacts with it full time and uses it to solve problems.
In summary, this post was a good exercise in getting my thoughts out of my brain and into some intelligible ideas. This has cleared up a lot of the way that I feel and has left me clearer in my position. This position is that lot of the reasons I hesitate are purely inside my head and something that I have to get over in my own time by doing lots of reading.
Thanks to: