A lot of people seem to struggle with the difference between the Selection tool and the Direct Selection tool (the black arrow and the white arrow respectively at the top of your toolbox).
In simple terms, the Selection tool (black arrow, hit 'V' on your keyboard to use) is for moving, rotating and scaling whole objects or groups of objects, while the Direct Selection (white arrow, hit 'A' on your keyboard to use) is for manipulating objects within a group, or individual points or segments of a line within an object.
But you can also use the Direct Selection tool to extract elements from a group or a merged object without ungrouping or releasing.
Assuming that you're using the ALT-Add Shape To Area version of the Pathfinder merge (which leaves the elements of the merged shape as individual objects, rather than creating one new shape) then you can pick up one element of a group with the Direct Selection tool, then cut and paste, which will remove the element from the group and leave the rest of the group intact, which is much quicker than ungrouping, selecting, moving or deleting an element, then re-selecting the remaining objects and re-grouping/merging.