The AND, OR and XOR are how to deal with pixels already on the screen. If you recall a pic (sprite, tilemap…) with AND logic, only pixels that are currently on (on the graph screen) and on in the pic will be filled in. With OR logic, a pixel that is currently on or is on in the pic will be filled in (this is how the regular RecallPic works). For XOR logic, a pixel will be filled in if [the pixel is currently on and if off in the pic], or [the pixel is currently off and on in the pic] (the [] are to break up those two statements).
Each pixel on the screen is either on or off, so it can be represented by a binary digit (bit). Eight of those together is a byte. The graph screen is broken up into a 96bit=12byte by 64bit=8byte array, with each pixel being represented as one of the bits. Thinking about it like this, ANDing a pic is the same as doing a bitwise AND of the entire current screen and the entire pic you're recalling. Likewise, OR/XORing a pic is the same as doing a bitwise OR/XOR of the entire current screen and the entire pic you're recalling.