My understanding is that Buddhists believe that the self does not really exist, but that reincarnation does. If the self does not exist, what is it that Buddhists believe is reincarnated?

Excellent question. The first thing to say is that there are many schools of thought within the world of Buddhist philosophy, and there are divergences of views within Buddhism on this question. I will give you two answers, each of which is adopted by a significant number of Buddhist philosophers. The second thing to say is that in the context of Buddhism, as opposed to other "orthodox" or "Hindu" schools of philosphy originating in India, it is better to talk of "rebirth" than "reiincarnation," since, as you point out, there is nothing that gets placed into another body. Now, when Buddhist philosophers say that there is no self, they mean that there is no single, continuing substance or subject of experience that remains the same throughout one's life, like a soul, or as Indian philosphical schools call it, an "atman." Instead, Buddhist philosophers argue, a person is consituted by, or posited as an entity based upon, a set of causal processes involving a physical body, sensations, perceptions,...

I was having an argument with a religious friend of mine and I told him that I didn't think I could argue with him anymore because his belief in God was irrational. He replied that my belief in reason was irrational. Is belief in reason irrational?

You friend is reviving a delightful argument from Sextus Empiricus (Outlines of Pyrrhonism) to the effect that any justification of the value of rational proof has to be either a rational proof and hence circular, or irrational and hence inconsistent with the conclusion one is trying to establish! It is a devilish argument. While it does show that you can't justify the use of rational argument from nowhere, it doesn't show that there can't be strong pragmatic reasons for using rational argument that do not consist in proofs that it is justifiable. The very fact that it is a deeply embedded and successful practice may make it prima facie justified, with the burden of proof for one who wishes to dislodge it on the other's shoulders. Note that critiques of rational argument are equally question-begging. Wittgenstein's ON CERTAINTY is a nice source for a subtle discussion of this issue.