The calculator can work with numbers written in three bases: decimal (the usual), binary, and hexadecimal. The 0h command indicates that a number is written in hexadecimal:
:0h10 16 :0h2F6 758
When written in hexadecimal, numbers are expressed as signed 32-bit integers (32 bits correspond to 8 hexadecimal digits), which means that only the integers between -231 and 231-1 can be expressed in hexadecimal. With other hexadecimal-related commands, numbers are simply truncated to fit in this range. Not so with 0h: if you enter more than 8 hexadecimal digits after the 0b, the result is a domain error.
Even if the calculator is in hexadecimal mode, you still have to write 0h for an integer to be interpreted as hexadecimal: the mode setting only affects output. If the calculator is in decimal mode, which is the default, you have to use ▶Hex to get output in hexadecimal.
260 - Domain error happens when 0h is used with more than 8 hexadecimal digits after it.