If you were pulling off the layers, then you weren't smoking the blend. From what I have heard, you have to slice "into" the layer, across the grain.
That would be like eating a pumpkin pie. Scraping off the whipped cream and eating it, then scraping off the top layer of pie, with some cream flavor, then eating the moist center, then eating the graham cracker crust. The pie was designed to be eaten by the slice, not by the layer.
The same with cake tobacco. I am sure the layers were good on their own, but the blend was absent. If the cake crumble easily, the best alternative would have been to crumble it all and rubbed it out, mixing the tobacco together so that when you fill the pipe, you have all of the layers present.