AI in Cybersecurity⌗

Ever since the ChatGPT was released on the 30th of November 2022, it was inevitable that AI would become a crucial part of our lives. Nowadays (even though its been only 3 years since it came out), we see AI used everywhere. We write code with AI, generate images and videos with AI, summarize our meetings with AI, and with the recent introduction of tools like Ghidra MCP, we are seeing its increased use in Cybersecurity. But is this entirely as evil as the recent situation on HackerOne makes it out to be? I wouldn’t be too quick to judge.

The HackerOne Bug Report⌗

Before I continue, I recommend checking out the actual bug report here. To quickly summarize, there is a critical bug in HTTP/3 capabilities of curl, which can be leveraged to corrupt the memory, hence leading to Remote Code Execution, or RCE in short.

In the initial stages of reading this report, all seems normal and it seems like a standard report for a high severity bug. But reading further, it begins to seem more and more suspicious (to say the least). Firstly the curl staff member mentions that the patch is not applicable here and asks a question, to which the reply looks SUSPICIOUSLY like it’s AI generated markdown. The user provides the steps to APPLY a patch, without actually providing the patch in question.

Some more discourse later, one of the other curl staff members drops a banger about this part of the report:

Analysis shows:
	- Return address overwritten
	- Stack recursion at ngtcp2_http3_handle_priority_frame

Staff comments:

There is no function named like this in current ngtcp2 or nghttp3. 
Please clarify what you talk about. Which versions of ngtcp2 and nghttp3 
did you find the problem in?

The function he was talking about, ngtcp2_http3_handle_priority_frame does not exist. This, reveals the fundamental problem with relying completely on AI to do the heavy lifting on a high-skill job.

So What’s the Issue?⌗

AI generated reports are nothing new. There have been many reports, made using AI over the years and so far, according to the founder of the curl project, Daniel Stenberg, no valid bugs have been reported by these so-called AI generated slop reports (read his full linkedin post here). It is not uncommon to run into situations where the AI hallucinates things that do not exist in order to fulfill the task given to it by a user.

This is a worrying trend as in the recent years, the number of these reports have only been increasing. The problem is that the people behind triaging and validating a submission are human, and, it takes time. Triaging a submission is no simple task and people put a lot of time into this. Simply spamming AI-generated reports on these platforms only makes this job harder, as:

• The report cannot be ignored in the off-chance that it reports an actual exploit.
• Producing these kinds of reports takes very little time and the staff will not be able to keep up with the speed of incoming reports, resulting in lesser security.

The curl founder called it a DDoS Attack against curl due to this. And, he’s not wrong.

What Causes this Issue?⌗

I think the major problem with all this, is rooted in the fact that many people, beginners and professionals alike, have begun to SUBSTITUTE their workflow with AI, as opposed to AUGMENTING their workflow with AI.

AI is not 100% accurate, it never has been nor will it ever be, as it stands. AI is trained to give answers that SEEM correct. Obviously, this is not intentional, rather it is a consequence of how the current methods of training these AI models work. At the end of the day, the AI is only doing it’s job. It is us, as humans, whose job it is to validate this before making a submission.

But is AI completely Evil?⌗

No, absolutely not. I am not going to pretend that I never use AI for my tasks. I do, and I think the benefits to having something like an LLM enhancing your workflow shouldn’t be slept on. However, the key distinction is that AI ENHANCES my workflow, it is not my ENTIRE workflow. What I mean is that say, for example, in a project, I won’t be vibe coding that whole project using some random LLM, I would rather be using it to write boilerplate code, find resources online, or as a last resort, suggest fixes to bugs in the code.

And I feel that a similar approach must be adopted by the wider community at large. I am not saying my approach to this is perfect, not at all, but rather the knowledge of Enhancement vs Substitution must be more widespread.

It is due to this same pitfall, that vibe coding (a term which I absolutely hate by the way), is gaining more and more traction, when in reality, you can simply learn to code, put in the hours, and write FAR better applications than an AI ever could. Even better, you could double that productivity, by then finally using AI to enhance your already existing workflow, making this endeavour all the more worth it.

Closing Thoughts⌗

This situation on HackerOne clearly demonstrates the flawed mindset that many people have started to adopt with regards to the usage of AI and a shift in this mindset is necessary if the usage of AI in high-skill fields is to be a net-positive. However, not all of this is bad, and if anything, we should learn from the pitfalls clearly demonstrated by this incident and continue to improve in our respective disciplines.

Have a nice day! :)