While I agree that it will be very difficult to enjoy pipe smoking if you worry about its negative effects while smoking, I don't think that there is anything wrong with your initial worry, as studies have indeed shown a correlation between pipe smoking and certain diseases. I had the same worries at first, which is why I had a very long warm up before I came to a point where I would consider myself a pipe smoker, rather than a dabbler. This happened this summer, two years after my first starter kit.
One of the reasons for this is that I grew up in a very anti smoking family, and knew that I would have to defend my choice. So I did a little research and in the only really useful study, that I found focusing on pipe smoking, the amount of bowls per day for each segment of people studied were very high (according to my expectation for smoking). The lowest segment was 1-3 bowl per day, while the highest was 13+ bowl per day. For some diseases the relative risk was the same across all segments, but for most, IIRC, the risk increased with number of bowls. I have settled at a point where I smoke on average one bowl a day, and this is a happy medium for me, solving the tension between enjoyment and worry. I am aware of the increased risk but know that the relative risk is propbably not much greater than other health concerns, such as diet, lack of use of sun screen, lack of movement, alcohol consumption and other elements. For me the relative risk is worth it, and that is what you have to decide for yourself.
Besides, AFAIK recent research shows that cancer is most likely genetically determined, which is why some people are so genetically disposed that they can smoke and drink an entire life and live into their 90es.