No, you can't ask whether your notification icon is visible or not
A customer asked what started out as a simple question, but by now you know that this sort of simple question only raises new questions:
Is there a way that we can detect that the user has hidden our notification icon?
No, there is no way to ask whether your notification icon is hidden or visible. Users decide which icons they want to see, and you don’t get to snoop in on the users’ decisions. Questions like this scare me, because it sounds like this customer’s program is going to respond to the user’s choice to hide their icon by doing something even more annoying. This is another demonstration of Le Chatelier’s Principle as interpreted by John Gall: Every system resists its proper functioning. In this case, a design to make programs less annoying and intrusive merely drives those programs to even more annoying and intrusive behavior. The customer’s response confirmed this suspicion.
We have a long-running process with multiple stages. It starts with a dialog box and a progress bar, and after the dialog finishes, a long operation takes place in the background. In some cases during this background operation, we need to ask the user to take an action or make a decision. Our previous version of the program displayed a dialog box under these circumstances, but we switched to a notification balloon to make the experience less annoying. But if the user has hidden our notification, then we want to go back to showing the dialog. (Ideally, we would make the long background operation part of the initial dialog box, but we do not control the initial dialog box.)
If you use a notification icon, then you have to accept that the user can hide notification icons. If you still want a notification icon, you can start by displaying a dialog box with a progress bar, and give it a button that says “Hide this dialog and display a notification when attention is required.” With this design, converting the dialog box to a notification icon is an active step the user must take, and if the user hides the notification icon, it’s their own fault.
Or abandon the notification icon entirely. According to the notification area design guidance, you shouldn’t use notification icons if immediate action is required.
Is the status critical? Is immediate action required? If so, display the information in a way that demands attention and cannot be easily ignored, such as a dialog box.
Windows 7 lets you communicate the progress of the background operation unobtrusively by displaying the progress bar as part of the taskbar icon.