Personally, I think you should ask the permission of those four people. If they are happy to stay with you and can afford one more person in the dormitory, then I support you to do that. Because I know very well how uncomfortable it is to stay in a very difficult dormitory.

But you should be prepared, maybe the dormitory you go to will be troubled by your joining, which is not a good thing for you.

My personal suggestion is to formally change the dormitory. Find a counselor and try to go to another better dormitory. If not, then protect your feelings first.

After all, four years in college is so short, it's not good to be unhappy every day, is it?