It seems that you could have reached this conclusion faster by elaborating on your use of the profiler. Don't assume that cycles are spent on instructions. Look at your IPC and drill down into what CPU-bound means for your workload. In your case I think a standard top down analysis would have made the virtual memory management cost jump right out.
Thank you for your suggestion. You are absolutely right, XD. In fact, the order of events in this blog post does not match the actual order in which I debugged and analyzed the issue.
After I identified the TLB misses and confirmed that huge pages were effective, I noticed that there were still many suspicious points in the flame graph. Interestingly, my agent attributed the effectiveness of huge pages to those suspicious points, which turned out to be unrelated to the bottleneck at the time. That sparked my curiosity.
The structure of this blog post was mainly chosen to make the story easier to follow, while also covering the various issues I investigated in depth along the way.
In fact, I recently switched from my previous job building data infrastructure on cloud to an HPC-related role, so I am still not very familiar with some of the mature practices and established conclusions in the HPC world.
So thank you very much for your suggestions. I also hope to learn about more and better methods that can help people identify root causes more quickly and accurately in complex scenarios.
This might be orthogonal to the TLB miss overhead you found, but have you looked at using P2PDMA to transfer directly from the NVMe SSDs to the NIC? Not sure how the CRC calculation would play into that.
Thank you for your reply. This is a long-running service. Without CRC validation, errors caused by partial writes could accumulate over time and affect correctness. Therefore, we adopted this approach.
Depending on your hardware you may be able to do the CRC validation in the NIC. If you're using Intel you could use DSA also, but then you're still copying the data through DRAM.
Just throwing out some ideas, obviously the best solution is the one that you already have working :)
Author here. This post is a write-up of a performance-debugging rabbit hole I hit while trying to saturate NICs with NVMe reads using io_uring and RDMA.
The short version: READ_FIXED fixed the obvious per-I/O GUP overhead in a small demo, but the larger deployment still got stuck at roughly half of line rate. After ruling out io-wq backlog, request splitting, fd lookup, and CRC arithmetic, the actual wall turned out to be dTLB misses from scanning 1,028 KiB buffers backed by 4 KiB pages. Moving the read arena to hugepages brought the system close to NIC saturation.
The funny part is that an AI agent suggested hugepages early and got the optimization right, but its explanation was wrong. This post is mostly about reconstructing the evidence for why it worked.
I’d be very interested in feedback from people who have used AI to debug performance issues in a complex system.
I disagree with the AI part. Because hugepages is one of the things that can be guessed to improve performance when doing something with substantial amount of data.
So anyone familiar with the space could have suggested something like that without knowing the details of the problem. Hence it is not useful advice IMO.
That aside, the blog post was really cool to read and a instant favorite, wish there were more english posts on the blog.
Especially like the hardware limit based expectations, detailed measurements and the writing style.
Thank you for liking this blog. I agree with your point. Actually, I’ve just recently transitioned from building data infrastructure on the cloud to taking on a high-performance computing role that truly handles massive amounts of data. So, although I’d heard about the benefits of hugepages before, I had never actually reproduced these issues in my own environment. This time, even though I initially suspected the problems were related to hugepages and the TLB, I didn’t write this blog from a seasoned perspective. Instead, I wanted to methodically investigate and eliminate all other possible issues I could think of. (Interestingly, my agent attributed the effectiveness of hugepages to the root causes of these bugs, which piqued my curiosity and drove my deeper exploration.)
Finally, thank you very much for your appreciation, which means a lot to me. Previously, I was working on open-source projects, but now that I’ve changed jobs, I may not have the same amount of energy to contribute to open-source code as before. However, I think blogging might be a new way for me to contribute. I hope I can keep it up.
(My English writing skills are poor, so I wrote in Chinese and used AI to translate it; I hope you don’t mind.)
There's no personal attack here. There's a message addressed to a person.
That is, your statement has no sense. If you believe it is, please report it. Oh, please put your public defamation and devaluation of someone's commentary together, considering it the same "personal attack" towards me.
If you are so afraid of receiving messages that address your name, please reconsider your presence on the Internet.
If you will invest your time and effort addressing me on my projects published, where I assume responsibility and my authorship of them, being accountable for the works I publish on public.
That is, it is a work of someone who published online, and that is a responsibility to be taken, hence licenses exist, and your name on it. You are free to be "personal" on my work, too. Please do! I would appreciate it, if that would help us discover more and educate ourselves on subjects involved.
It was quite personal (e.g. "your time you waste on sloppery slops instead of actually reading") and quite aggressive. Moreover, you've been posting like this in other threads as well:
If you say you didn't intend these as personal attacks, of course I believe you. But such posts are the kind of thing we ban accounts for, so if you want to keep posting to HN we need you to stop this.
I have no idea why you support slop and hate on people effort, including me trying to highlight a thing being a slop where no people actually worked on their project but just paid for a few tokens, wrote a prompt, and published the model output.
It's sad you negate all my other effort, and fixate on these few comments where I share my stand for my beliefs for human work, experience, effort, and love... And it's also sorrowful you list just a few references to my messages in your negative accent, here for everyone to shame on whatever you believe yourself. You should list my other contributions and comments (which you just ignored and dimmed) to judge my presence and history fair, human, and responsibly.
Slop? Never, and I would rather die. Meanwhile, I just so heartfelt hope the agent-circled person will re-consider their time and life, and listens to the screams and sorrow inside the datasets of these LLMs which are built on stolen by vendors (e.g., OpenAI, Anthropic, High-Flyer, Mistral, Cohere, Google etc.) effort and art of now unknown/defaced artists and developers...
If you have problems with me, I ask you to discuss this in private. Otherwise, your attitude does not seem professional or accountable.
I appreciate you heartfelt... for your marvelous effort, time, and love... you've already invested into the platform. Yet, it's obvious that the platform has changed. It's sorrowful to know Hacker News became this utter mess, where also initially trusted people derail conversations and threaten members on public, and for this? Thankfully, there are other venues, including public even as Lobsters (previous Barnacles), IndieHackers, DailyDev, DevTo, Tildes, HackerOne, or fediverse-like Lemmy/PieFed instances even.
Lies lies lies
Mods routinely break their own policies. You delete accounts that do not break any guidelines, you delete accounts because they say things you don't agree with, lies lies lies.
You allow government agents to do whatever they want on this site. Lies lies lies. You're so full of yourself too it's gross. Liar.
This sounds like a strong statement with little backing. The author does infra at DeepSeek if his LinkedIn is to be trusted, and is the author of Foyer.
You are the dumbest developer in the current space. Just because you can't self host and rely too much on provider's subscription, doesn't mean others follow the same.
Local self hosted models are the future and no one can take that away from you. Get that through your little brain.
