Hi,
How did you remove the dialog instance . Did you use SWPM(sapinst) and then go to Software lifecycle management, there you will find the option to uninstall the instance, it will ask you for he instance SID as well as Instance number too, then it will remove all the registry entries as well as files.
can also check the below note...
1259982 - UNIX:Deleting an SAP System Based on NW 7.1 and Higher
SAP Note 871523 - Removing dialog instance from database
Thanks
Rishi Abrol