But why do we need the bitwise AND for that, specifically? I understand the idea, but would it not be easier to only parse the IP address string of bits only for the first n bits and then disregard the remainder (the host identifier)?

Essentially it boils down to:

bit operations are stupid fast and efficient, String operations are super slow.

Also, IP addresses are always stored as int32/int64, so applying String operations would require them to be converted first.

Probably because it’s only four bytes of data, and counting/extracting bits takes more cpu time than one AND operation.
Most CPU’s are optimised to work with whole integers (32/64 bit) rather than individual bits.

If memory was a serious concern you could compress it down to one byte as a ‘number of 1s’ counter at the cost of additional cpu operations, but because 3 extra bytes is such a small amount of data, this memory/time trade off isn’t worth it in most systems.

It’d be useful if you wanted to compress some data logs or something with many subnet masks though.

I'll address your question in two parts: 1) is it redundant to store both the IP subnet and its subnet mask, and 2) why doesn't the router store only the bits necessary to make the routing decision.

Prior to the introduction of CIDR -- which came with the "slash" notation, like /8 for the 10.0.0.0 RFC1918 private IPv4 subnet range -- subnets would genuinely be any bit arrangement imaginable. The most sensible would be to have contiguous MSBit-justified subnet masks, such as 255.0.0.0. But the standard did not preclude using something unconventional like 255.0.0.1.

For those confused what a 255.0.0.1 subnet mask would do -- and to be clear, a lot of software might prove unable to handle this -- this is describing a subnet with 2^23 addresses, where the LSBit must match the IP subnet. So if your IP subnet was 10.0.0.0, then only even numbered addresses are part of that subnet. And if the IP subnet is 10.0.0.1, then that only covers odd numbered addresses.

Yes, that means two machines with addresses 10.69.3.3 and 10.69.3.4 aren't on the same subnet. This would not be allowed when using CIDR, as contiguous set bits are required with CIDR.

So in answer to the first question, CIDR imposed a stricter (and sensible) limit on valid IP subnet/mask combinations, so if CIDR cannot be assumed, then it would be required to store both of the IP subnet and the subnet mask, since mask bits might not be contiguous.

For all modern hardware in the last 15-20 years, CIDR subnets are basically assumed. So this is really a non-issue.

For the second question, the router does in-fact store only the necessary bits to match the routing table entry, at least for hardware appliances. Routers use what's known as a TCAM memory for routing tables, where the bitwise AND operation can be performed, but with a twist.

Suppose we're storing a route for 10.0.42.0/24. The subnet size indicates that the first 24 bits must match a prospective destination IP address. And the remaining 8 bits don't matter. TCAMs can store 1's and 0's, but also X's (aka "don't cares") which means those bits don't have to match. So in this case, the TCAM entry will mirror the route's first 24 bits, then populate the rest with X's. And this will precisely match the intended route.

As a practical matter then, the TCAM must still be as wide as the longest possible route, which is 32 bits for IPv4 and 128 bits for IPv6. Yes, I suppose some savings could be made if a CIDR-only TCAM could conserve the X bits, but this makes little difference in practice and it's generally easier to design the TCAM for max width anyway, even though non-CIDR isn't supported on most routing hardware anymore.