[P34] Are XGBoost and XGBoosting.com all we need?

Yes, it seems so.

I like to learn, preferably one and, at most, a few things very well, and apply them to most problems. This way, I get to iteratively learn these few things deeply by referring to them repeatedly. For tabular data, some very experienced people in the field of ML believe that XGBoost is all we need, and they obsess over learning to use this algorithm in various ways.

I have been seeing the “XGBoost is all we need” meme for quite some time, and I thought covering the whole landscape of tabular ML using this one algorithm would be a great way to learn both the algorithm and the field. This is partly why I have been using only XGBoost in my blog series to represent tree-based algorithms. The other reason is that XGBoost and LightGBM are the only two GBT algorithms I have ever used for real-world ML work.

In addition to preferring to learn one or a few things, I also like to learn from one or a few sources so I can be aware of most nuances of the topic the teacher knows, rather than following shallow, disconnected sources. Today, I was looking for a comprehensive resource on XGBoost that I could refer to, and I found this website XGBoosting.com by Jason Brownlee, Ph.D (AI).

I first came across his work in 2024 through his other website, Machine Learning Mastery, and have been referring to it since then to look up an explanation or implementation. I quickly went through XGBoosting, and it looks like a very good resource for learning XGBoost and its use cases in a structured manner through bite-sized blog posts. So you might find me referring to this resource while talking about XGBoost.

With this, I think XGBoosting, Machine Learning Mastery, and Christoph Molnar’s work on ML cover a lot, if not most, of what one needs to learn the field deeply. Now that we have tabular foundation models that outperform XGBoost, XGBoost and TabICL (perhaps not TabPFN because of its license) might be all that most people need for tabular ML.