It started as a hardware problem and doesn't seem to be slowing down. LTE needed more and larger antennae for lower frequencies than older tech. Four cellular antennae are now pretty standard. Then you have wifi, Bluetooth (which can share if they can TDM), wireless charging, NFC, ultra wideband, GNSS. Then the chips are so powerful they need heat dissipation systems installed (or just lame thermal throttling like what Apple does.)

The modems require more power, (especially at the beginning of LTE) which means bigger batteries. LTE and NR have reduced range compared to the older narrowband technologies, so the phone needs to use more power to transmit, especially when carriers like Verizon didn't backfill cell sites to compensate for the reduced coverage.

Then, cameras, one wasn't enough, 4 or 5 are very common now (usually 3 primary and depth or low res sensors for aiming.)

When tablets became popular, many people decided to just have a large phone screen rather than a tablet, further entrenching the size.

The tech is more mature now, a 2-antenna MIMO antenna for cellular would suffice, albeit at the expense of network performance. Likewise one camera with a depth sensor would work, although mobile photography would be more limited. Dropping some limited-use items like wireless charging and ultra wideband could further shrink space.

So it would be possible now, but as others here have mentioned, the supply side focuses on larger hardware.

Ironically, at this point I'd almost prefer a smart watch with LTE and stop carrying a phone altogether. However, the aforementioned antenna issue makes it so watches generally have poor to unusable signal, poor battery life in cellular mode, no camera, and the 5G NR low power spec/chips aren't fully done yet, so it's LTE only on them, which, with carriers transitioning to 5G will make it so watches can only access a handful of congested bands.

Also, that device manufacturers tend to design smart watches to be companion devices to a smartphone rather than primary makes that concept's execution problematic.

Another idea I had that was anti small phone but huge battery boost was to just bring a backpack or a satchel or whatever. Carry a full sized tablet around, and use a Bluetooth headset for calls. However, tablets are also often crippled by carriers/manufacturers so they can't do common things like SMS or voice calls, and Apple has basically monopolized that market.