Probably the bad guys are killing and competing with your e.coli and bifidobacteria. Maybe an "overflow" of this good guys (e.coli and different strains of bifidobacteria -b.bifidum,b.longum,b.infantis ...) can help. When you kill enought you can lower the dosage very slowly so your own e.coli and bifidobacteria start to growth (keep taking FOS).
Also bacillus subtilis (discovered by the Nazi German medical corps in 1941) may be useful. They use it against the pathogenic bacteria that causes dysentery.