Do i believe it is only damaging to poor people? No.
Do i believe that the education system was designed to cater to middle class expecatations? Yes.
As it happens my solution is to look at the cause of the problem, it comes down to some fundamentals of human nature. You look at the spikes in violence among male teens in poverty stricken inner city areas, and you see a natural tendancy to compete, the same thing in better off schools, except now the competition is academic.
The difference is socially conditioned values.
Humans will compete for status, and currently status is associated tigthly with income - however income inequality has left an underclass who don't see the point in competing; who presumably see the sustem as being rigged against them, and thus reject it. They still compete with their peers, in the only way they see as valid (female teens have a similar spoke in sexual activity)
We can break this strong link between income and status by creating a universal basic income; more than that we can build communities where people gain a sense of belonging and feel they can acomplish something (and gain status and respect within) which are driven by voluntary contribution of the members.
But creating communities is not something you can solve with top down dictate (like the communists tried) you need bottom up organisation (actually, the kind of grass roots efforts which bernie sanders is driving his campaign with) Government needs to createthe environment where people can thrive. And then let the people get around to deciding how that thriving looks.
The education system in particular should be reformed with a combination of independent learning and student chosen topics. Teachers are not the font of all knowledge, and should be seen as guides, motivators, and experts in how to learn - not subjects. We have the technology, we have the likes of KhanAcademy, and tonnes of educational resources available for free on youtube; the old system is obsolete. Students should want to learn, and the older cirruculum was designed without knowledge of what the future holds. Kids should be given more reaponcibility for their learning, and more power, so that they love it. Healthy competition should be encouraged, without the associated 'failure' in the eyes of society (a society which doesn't have a clue what skills will be needed in the rapidly changing future, which doesn't teach programming or social skills, despite these being two of the most important in the economy...)
That would be where i would start at least.