Aye, that's a bit of a problem for me. What I think is meant by "brother" and "bretheren" and "father" differ somewhat from what many (not all) Christians have told me about them. (No one's really mentioned "neighbor" before -- I think we all kinda get the idea that the only relationship indicated there is mere proximity of domicile...)
In the original languages of the time (as I understand it -- linguists might have alternatives to offer), a "father" was one who possessed a characteristic. A man with a long nose might be called the "father of a long nose". A generous man might be the "father of generosity". So the word "father", as used back then in Jesus's time, had a secondary but more familiar meaning, not unlike how we use the word "fathers" as in "founding fathers (of the United States)".
"Brothers"... what was its familiar meaning back then? Did Jesus use "brother" to mean "fellow Christian"? I doubt he was preaching to a lot of Christians back in his day, so it seems unlikely to me that he meant brothers exclusively in this "saved" context. If God is the "father" of all -- and you'll get no argument from me on that point -- then we are all necessarily "brothers" (and "sisters"), regardless of whether or not we've accepted Christ.
I'll grant that in many instances, Jesus is quoted as speaking to his disciples, in which case a more familiar, more closely-bonded meaning could apply, but I don't often see it that way when I read the words.