Both are made from the linkage of repetitive units. In the case of polymers it is the repetition of monomer units (Styrene, ethylene, glycol, etc.) and for silica, it is the repetition of SiO4. Therefore the polymer has an organic backbone whereas the silica gel has an inorganic one. The polymer is more susceptible to leaching which happens when the polymer condensation is not completed and parts of it can dissolved in organic solvents.
The polymer active sites are inside the matrix so the rate of action is largely dependent on the rate of diffusion through the polymer. Polymer must be used in solvents that will allow it to swell (DCM, THF, CHCl3). This also means that product can get trapped inside the polymer, greatly affecting the yield.
The silica active sites are on the surface where they are accessible giving fast kinetics and high yields. Silica gel is not affected in any way by any organic solvent since the pore structure is rigid, permanent, and consequently is mechanically and thermally stable.