People are making the (valid) assumption that LE is far more common than BE.
On x86 swapping endianness is one instruction (BSWAP), but AFAIK it can't be SIMD'd easily. Given that SIMD is one of the things that can greatly speed up image processing, I don't think it "won't hurt performance too much"...
(Just FYI, this only works with SSSE3+, which to be fair is essentially all x86 machines. It's also >1 cycle on mobile x86 chips, which is very frustrating.)
On x86 swapping endianness is one instruction (BSWAP), but AFAIK it can't be SIMD'd easily. Given that SIMD is one of the things that can greatly speed up image processing, I don't think it "won't hurt performance too much"...