SSIS Remove redundancy

Here comes one more challenging yet interesting topic to tide over. The requirement goes something like this: You have some sources, let it be some sales data or some Call center data coming from different sources. Data can be of different media. But you are able to load it in your staging tables i.e. one staging database you are maintaining like StageDB for storing these incremental data.

So, here we have both master i.e. Dimensional data and Detail i.e. Fact data in our stage database. But before processing further and loading it into our Mart or Data warehouse, we need to check if there is any redundancy at the row level for each of these tables in the staging database. Here comes the main problem, what if you don't know how many tables are there in the staging database and how many columns are there for each table in the staging database but still you have to keep only unique records for each table.

This means there must be some mechanism to deal with this problem dynamically, i.e.

It has to detect the number of tables for the StageDB database.

It has to detect all the columns for each table in the StageDB database.

It has to detect only the redundant records for each table in the StageDB database.