If asked to sum this up quickly, I'd say that the M4 was the better tank, but the later Pz IV had the better gun, and as a result no version of the M4 could afford not to take the Pz IV seriously.
The remarkable thing about the Pz IV was that in 1945, when nearly a decade old, it was still a threat to the tanks designed to beat it. Thanks to that 75mm L48 there were very few allied AFVs around in 1945 that a Pz IV couldn't kill, even if it was itself pretty vulnerable at most battle ranges. You can't really say the same for the M4...
None of the allied mediums were designed around killing or beating the Pz IV. Armament and chassis modifications (I don't count tank based SPGs as new vehicles) were all in a response to much heavier German vehicles, much in the way German armament and chassis modifications were in response to the unexpectedly heavy Soviet tanks.
Both the T-34 and Sherman were designed around respective medium tank doctrinal demands and based on expected combat conditions. Sure Pz-IV is part of that condition package. Certainly T-34 owes one hell of a lot more to Pz-III and bits from various other designs and the perceived doctrinal need for a universal tank, than anything to do with what was, at the time an infantry support tank.
I also have the reverse opinion of how much better follow on designs should be. As a gross generalization (with plenty of counter examples) engineering capability was pretty even internationally. There is only so much you can do in terms of raw capabilities of vehicles (ie armor protection, horsepower, overall mobility) for a given weight class. Where there is huge room for improvement is in ergonomics: ability to use the thing effectively, reliability, ease of maintanance, ease of production, etc, etc, etc, etc.
By late war basically everyone's mediums could kill (at least at some range and not necessarily frontally) most other tanks on the battlefield. It's also a bad idea to conflate vehicle design with armament design. The latter is basically a totally seperate field and the two only overlap in fitting the cannon and storing all it's ammo.